English · 简体中文 · 🚀 Usage · 📊 Results · 🧭 Roadmap · 📖 Levels contract

Jiamu Zhang¹     Tianze Yang¹     Yucheng Shi²     Liang Wu¹

_{¹ Nokia, Sunnyvale, CA      ² Tencent Hunyuan}

_{Qwen3-8B on a real BANKING77 item. Every number is a model output.}

[!TIP] 🆕 L2 has landed. A closed-form head per question, solved on 100–300 labels in seconds, served from one prompt stopped at two thirds of the model's depth. It follows its question across rewordings without new labels. Jump to it ↓

✨ What it does

Ask any open LLM a typed question and get back a decision with a probability you can threshold, read from one prefill of its next-token distribution. No generation, no parsing, no fine-tuning. Raw logits change their answer when you reorder the options, and their confidence cannot be trusted; AnyJev fixes the first with zero labels and the second with a few hundred.

The last row is the point. Accuracy moves by 6 points, but the share of traffic you can safely automate goes from 7.7% to 52.0%, a 6.8× difference on this task (a point estimate at n=300; the interval is wide, see Limitations). With raw logits a "0.9" is not trustworthy enough to act on, so everything goes to a human. Once the probability means what it says, you can set a threshold.

🚀 Usage

📦 1. Install

pip install "anyjev[hf]"

💬 2. Ask typed questions. L0 is on by default and needs no labels.

from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend

d = Decider(HFBackend("Qwen/Qwen3-8B"))

route = Question.choice("Which team should handle this?", ["billing", "technical", "sales", "other"], name="route")
risky = Question.noul("Is this tool call destructive or irreversible?", name="risky")
done  = Question.score("How complete is the task?", bins=5, name="done")

r = d.decide({"conversation": [...], "tool_call": {...}}, [route, risky, done])
r["route"].distribution    # {"billing": 0.81, "technical": 0.07, ...}
r["risky"].p_true          # 0.12
r["done"].value            # 0.35
r.level                    # "L0"

🎯 3. Add labels when you have them. A temperature is L1; a closed-form head is L2, the accurate one.

d.calibrate(risky, states, labels)          # 100–500 labels → L1 (a temperature)
d.fit_head(route, states, labels)           # 100–300 labels → L2, one forward + a closed-form solve, seconds
d.save_artifacts("qwen3-8b.json")           # d.load_artifacts(...) next time; ~100 KB per head

r = d.decide(state, [route], level="auto")  # L2 where a head routes, else L1, else L0
r["route"].level                            # "L2"

🔁 4. Or let the loop feed it. d.observe(route, state, label) stores labels as they arrive and solves the head by itself at 30, re-solving at 60, 120, …

⚡ Serving. The transformers backend (anyjev.backends.hf) serves every level today; serving through vLLM / SGLang is on the roadmap, not in this release. For many states and one question, d.decide_batch(states, question).

🎬 Try it in one command. python -m demo.jev_mode --backend fake runs the whole thing on a synthetic model in under a second, no download. --lifecycle plays the deployment loop; drop --backend fake to run a real Qwen3 with the shipped heads (demo).

🧠 How it works

Every Decision carries its level, so downstream code can refuse to act on the wrong one. L0 costs K prefills for a K-option choice (about 0.25 s per decision at batch 32 on one H100, K = 20); L2 costs less than one plain forward — one prompt, stopped early: 0.68× on Qwen3-8B.

🔁 A head that maintains itself

L2 is not a training run. Labels buy a head in one closed-form solve (seconds on a CPU, no gradients, the model's weights untouched). After that only the head's feature mean and scale move, re-estimated from unlabelled traffic — so the head follows its question across rewordings and option orders by itself, and new labels are needed only for a new question.

Reworded, the Qwen3-8B head as is drops from 0.77 to 0.65–0.70; 30 unlabelled requests of the new wording bring it back to 0.74–0.75, against 0.77 for a fully relabelled refit (JSON).

One decision at serving time. A stored head answers from one truncated forward. Without one, the same call falls back to L1 or L0 exactly as before; the routing is in docs/method_v3.md.

Deployment lifecycle: day 0 at L0, labels from the loop, heads in seconds

flowchart LR
    D0["day 0: define the questions,<br/>serve with level auto;<br/>every answer is L0, zero labels"] --> C["collect labels from the loop:<br/>review queue, outcomes, or the LLM<br/>being replaced; dec.observe fits at 30"]
    C --> F["fit_head per question;<br/>export_artifacts to one JSON per model"]
    F --> S["serve: L2 where a head routes,<br/>L1 or L0 elsewhere"]
    S --> W{"what changed?"}
    W -->|"wording or option order"| S
    W -->|"new question or option set"| C
    W -->|"new base model"| R["re-solve every head from<br/>the stored labelled states"]
    R --> S

    classDef shipped fill:#dcfce7,stroke:#0f9d76,color:#0f172a
    classDef decision fill:#fef3c7,stroke:#d97706,color:#0f172a
    class D0,C,F,S,R shipped
    class W decision

A shift in the states (not the wording) is invisible to the recentring, so a periodic spot check on a labelled slice stays in the recipe. Full method: docs/method_v3.md.

📊 Results

Jev mode, on LocalLLaMA/typed-decisions (20 questions, 300 labels each, 2,000 held-out decisions):

A 1.7B at 64% of its depth reaches the number Jev publishes; a 4B ties the fine-tuned 421M Laya. 100 labels already put the 8B head at 0.740 (20 labels: 0.654, 300: 0.772).

_{More: Jev mode in full · 2048 and Minesweeper · NanoJev maze · when L0 helps · small models · the research log, negative results included}

_{Every number is regenerated from committed JSON (bash scripts/regen_docs.sh); a second run from a clean checkout reproduced every zero-label number bit for bit. Not affiliated with TypeSafe AI or Jev; rows published by their authors were not rerun here.}

🧭 Roadmap

• choice, noul and score from one prefill, nothing generated
• L0 with zero labels; L1 artifacts as JSON; levels enforced with require=
• L2: a closed-form head per question, routing, label-free adaptation, level="auto", observe
• Shipped heads for five Qwen3 models; a packaged demo (python -m demo.jev_mode)
• L2 on served engines (vLLM / SGLang): the residual stream at one block, or a truncated checkpoint
• Agent-loop evaluation: the same decisions inside a real agent, against the LLM they replace
• Heads on the Hugging Face Hub, an interactive Space, a technical report
• More models (Llama, Gemma, Mistral, DeepSeek), span readout beyond 26 options, conformal abstention

Dated plan and help-wanted files: ROADMAP.md.

🔍 Limitations

• On typed-decisions, "accuracy" is agreement with a teacher LLM. The gold is the mean of three samples of one model; a fresh sample of that teacher agrees with it 0.735 of the time.
• L2 is per question and per model. Heads fit on other questions do not help a new one, and only Qwen3 heads ship. It also needs hidden states: transformers today; vLLM / SGLang are on the roadmap.
• Calibration cannot fix a model that cannot answer. On maze edges and Minesweeper no readout beats the trivial baseline.
• L0 is not a free win everywhere. The batch prior costs accuracy when one label dominates (when L0 helps).

_{Also: at most 26 options in the letter readout (a span readout is on the roadmap, not in the code); coverage at 5% risk is a high-variance estimate at n = 300; the headline tables are Qwen models; every decision here is scored in isolation, not inside an agent loop.}

🤝 Contributing and citation

Backends and bench providers are one file each; several are help wanted (ROADMAP.md, CONTRIBUTING.md). Changes: CHANGELOG.md. Credits: CREDITS.md.

@software{anyjev2026,
  title  = {AnyJev: Turn any LLM into a Jev-style decision model},
  author = {Zhang, Jiamu and Yang, Tianze and Shi, Yucheng and Wu, Liang},
  year   = {2026},
  url    = {https://github.com/nokia-applied-research/AnyJev}
}

Apache-2.0, see LICENSE. Datasets keep their own licenses, see THIRD_PARTY.md.