jeff

A self-hosted implementation of TypeSafe's jev System One API, powered by GLiFormer (400M parameters). Use the official typesafe-sdk by pointing TYPESAFE_BASE_URL at jeff.

Supports choice (pick an option), score (rate on ordered levels), and noul (probability of yes). Cheaper to self-host, but less accurate than jev on reasoning-heavy tasks. See benchmarks.

Quickstart · Deploy · Configuration · API · Development

Quickstart

Requires uv and Python 3.12. Run from the repository root:

uv sync --extra dev
uv run hf download knowledgator/gliformer-large-v1 --local-dir models/gliformer-large-v1
JEFF_API_KEYS=devkey uv run jeff

Serves at http://localhost:8000. Device selection: CUDA → MPS → CPU.

The sync above also installs typesafe-sdk. Save this as example.py:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient(api_key="devkey", base_url="http://localhost:8000")
result = client.system_one(
    "I was charged twice. Please help ASAP.",
    {
        "billing": Noul(instructions="Is this about billing?"),
        "tone": Choice(
            instructions="What is the tone?",
            criteria={"calm": None, "angry": "hostile"},
        ),
        "urgency": Score(
            instructions="How urgent is this?", criteria=["low", "medium", "high"]
        ),
    },
)
print(result.nouls["billing"].noul)
print(result.choices["tone"].choice)
print(result.scores["urgency"].score)

In another terminal:

uv run python example.py

For an existing SDK app, set TYPESAFE_API_KEY=devkey and TYPESAFE_BASE_URL=http://localhost:8000 instead of passing client arguments.

curl http://localhost:8000/v1/systemone \
  -H 'Authorization: Bearer devkey' \
  -H 'Content-Type: application/json' \
  -d '{
    "state": "The export button crashes in Safari.",
    "model": "jev-latest",
    "questions": {
      "severity": {
        "type": "score",
        "instructions": "How severe?",
        "criteria": ["cosmetic", "degraded", "blocking"]
      }
    }
  }'

Deploy on Modal

Use L4 for the HTTP API. These commands download weights once and deploy with one warm container:

uv run modal setup
uv run modal run deploy/modal_gpu.py::download
JEFF_GPU=L4 JEFF_API_KEYS=k1 uv run modal deploy deploy/modal_gpu.py

For an ephemeral URL with no warm container:

JEFF_API_KEYS=devkey uv run modal serve deploy/modal_gpu.py

Deploy settings: JEFF_GPU=L4, JEFF_MIN_CONTAINERS=1, JEFF_MAX_CONTAINERS=8, JEFF_MAX_INPUTS=64, JEFF_TARGET_INPUTS=16. Server JEFF_* variables are forwarded. The GPU image defaults to batch size 32, batch wait 10 ms, and warmup enabled.

Measured HTTP throughput caps at ~50 requests/s per container; scale containers for more. A10G performs better for direct backend calls and long requests. See results.

uv sync --extra onnx
uv run python scripts/export_onnx.py models/gliformer-large-v1 --int8
JEFF_BACKEND=onnx JEFF_QUANT=int8 JEFF_THREADS=8 JEFF_API_KEYS=devkey uv run jeff

uv run modal run deploy/modal_cpu.py::export
JEFF_CPU=8 JEFF_QUANT=int8 JEFF_API_KEYS=k1 uv run modal deploy deploy/modal_cpu.py

Configuration

Set environment variables before starting the server.

API and compatibility

Errors: 401 invalid key, 422 validation or request limit, 429 rate limit (retry-after-ms), 529 full queue. Responses include x-typesafe-request-id, x-jeff-server-ms, and x-jeff-batcher-ms.

The wire format works with the official SDK; model behavior differs:

• Probabilities: normalized sigmoids, temperature-scaled at 3.2. score uses the raw distribution, so it only matches the weighted average of displayed probabilities at JEFF_TEMPERATURE=1. Confidence uses (p_max - 1/n) / (1 - 1/n).
• Question independence: nouls get separate encoder passes; choice and score questions share a pass and can affect each other. Set JEFF_ISOLATE=all for independence at extra cost.
• Tokens: usage.input_tokens counts DeBERTa prompt + text tokens; output_tokens is nominal. Counts are not comparable to jev billing.

Benchmarks

Measured on 1,600 labeled items across eight datasets:

jeff is close on binary sentiment, tied on emotion classification, and substantially behind on irony, reading comprehension, and JevBench's judge and hard tiers. On JevBench its rank comes from cost; it is #14 of 18 on intelligence. Costs depend on workload and utilization. Full results, methodology, and reproduction commands → · JevBench leaderboard →

Development

uv sync --extra dev
uv run pytest -q

Model integration tests need models/gliformer-base-v1; they skip if it is absent. The SDK tests use a live server with a fake backend.

Code: core · backends · server · deploy · bench

License

MIT. See LICENSE.