{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-Step Decision Router: Read the Model's Answer Before It Speaks\n",
    "\n",
    "**Route messages with an LLM in one forward pass, then find out whether its confidence deserves your trust.**\n",
    "\n",
    "This is the companion notebook to the [DiamantAI](https://www.youtube.com/@DiamantAI) video **[The AI That Knows the Answer Before It Speaks [Jev Explained]](https://www.youtube.com/watch?v=Fo2kisJx92Y&list=PLBrpE2PttR2k)**. The film follows one fraud report through a chatbot model and through Jev, then runs the same test on about three thousand real bank messages. This notebook lets you run it on yours.\n",
    "\n",
    "## Overview\n",
    "\n",
    "Many agent steps are decisions, not writing: which team gets this ticket, is this comment spam, does this tool call look safe. Most pipelines still ask a chat model to *write* the answer and then parse the text. That is slow, it costs output tokens, and it throws away the one thing you need to act safely: how sure the model was.\n",
    "\n",
    "Models built only for this kind of decision have started to appear. TypeSafe's **Jev** is one: it returns a choice with a probability and never writes text. Jev is closed, so this notebook does the same move on an open model:\n",
    "\n",
    "1. **Read the decision at the first step.** Before an LLM writes its first token, it has already scored every token it knows. Read the scores of your options and you have the choice plus a probability, in one forward pass.\n",
    "2. **Compare it with the chatbot way.** Let the same model think out loud and write its answer, then compare time and accuracy.\n",
    "3. **Check whether the number is honest (calibration).** Group answers by confidence and count how often each group is right.\n",
    "4. **Fix it and put a gate on it.** Rescale the confidence on held-out data (temperature scaling), then route automatically only above a threshold and send the rest to a person."
   ],
   "id": "cell-0"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📖 **Going deeper:** Chapter 1 of [*AI Agents Made Simple*](https://europe-west1-rag-techniques-views-tracker.cloudfunctions.net/rag-techniques-tracker?notebook=all-agents-tutorials--one-step-decision-router&click=book-nb-agents-top&target=https%3A%2F%2Fwww.amazon.com%2Fdp%2FB0HNXRSRH2%3Fmaas%3Dmaas_adg_api_580351185840460851_static_9_573%26ref_%3Daa_maas%26tag%3Dmaas%26aa_campaignid%3Dagents-en%26aa_adgroupid%3Dgithub-notebook%26aa_creativeid%3Dgenai-decision-router-top&retarget=0&text=book-nb-agents-top), \"Do You Even Need an Agent?\", covers this topic. The book is by the author of this repository."
   ],
   "id": "13310eb7"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detailed Explanation\n",
    "\n",
    "### Motivation\n",
    "\n",
    "A routing agent that is wrong 3 times in 10 is fine *if it knows which 3*. Then those cases go to a person and everything else runs untouched. That only works when the confidence number means what it says: when the model says 90%, it should be right about 9 times in 10. A model with that property is **calibrated**. Language models are often far more confident than they are right, so you have to measure it on your own data before trusting it.\n",
    "\n",
    "### Key Components\n",
    "\n",
    "- **One-step decision:** one forward pass, then the log-probabilities of the *first token* of each option name, normalised over the options.\n",
    "- **Written baseline:** the same model with thinking turned on, generating a full answer that we parse.\n",
    "- **Calibration check:** a reliability table (confidence bucket → accuracy) and Expected Calibration Error (ECE).\n",
    "- **Temperature scaling:** one number `T` fitted on held-out labelled messages that softens or sharpens the scores.\n",
    "- **Confidence gate:** the lowest threshold that keeps held-out accuracy above your target. Messages above it route automatically, the rest go to a human.\n",
    "\n",
    "### Agent Architecture\n",
    "\n",
    "![One-Step Decision Router](../images/one_step_decision_router.svg)\n",
    "\n",
    "### Benefits\n",
    "\n",
    "- **Fast and cheap:** one forward pass and no generated tokens, instead of hundreds.\n",
    "- **Actionable:** a probability your code can branch on, not a sentence to parse.\n",
    "- **Honest:** the calibration check tells you, in numbers, when the model's confidence can be used.\n",
    "\n",
    "### Data\n",
    "\n",
    "[banking77](https://github.com/PolyAI-LDN/task-specific-datasets) (PolyAI, CC-BY-4.0): 3,080 real customer messages to a bank, each labelled with one of 77 intents. We map the intents to 7 teams, so the task looks like real ticket routing."
   ],
   "id": "cell-1"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Required Packages\n",
    "\n",
    "The notebook runs an open model locally with Hugging Face `transformers`. A GPU (Colab T4 is enough) or an Apple-silicon Mac is recommended, and a CPU also works, only slower. No API key is needed."
   ],
   "id": "cell-2"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q torch transformers accelerate pandas matplotlib"
   ],
   "id": "cell-3"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "### 1. Configuration\n",
    "\n",
    "`Qwen/Qwen3-1.7B` runs on a laptop or a free Colab GPU. The film used `Qwen/Qwen3-8B`, which is more accurate but needs about 16 GB of memory. `N_MESSAGES` keeps the first run short, and you can raise it to 3080 for the full test set."
   ],
   "id": "cell-4"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_ID = \"Qwen/Qwen3-1.7B\"   # the film used \"Qwen/Qwen3-8B\"\n",
    "N_MESSAGES = 400               # up to 3080\n",
    "N_WRITTEN = 5                  # messages to also run the slow, write-it-out way\n",
    "SEED = 7"
   ],
   "id": "cell-5"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load real customer messages\n",
    "\n",
    "We download the banking77 test split straight from the dataset's GitHub repository."
   ],
   "id": "cell-6"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io, random, urllib.request\n",
    "import pandas as pd\n",
    "\n",
    "URL = \"https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/master/banking_data/test.csv\"\n",
    "df = pd.read_csv(io.StringIO(urllib.request.urlopen(URL).read().decode(\"utf-8\")))\n",
    "df = df.sample(frac=1.0, random_state=SEED).reset_index(drop=True).head(N_MESSAGES)\n",
    "print(len(df), \"messages\")\n",
    "df.head()"
   ],
   "id": "cell-7"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Map 77 intents to 7 teams\n",
    "\n",
    "Real routers send a message to a team, not to one of 77 intents. This mapping is ours. Some messages could fairly go to two teams, and that matters later: an honest model should *split* its probability on those."
   ],
   "id": "cell-8"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEAMS = [\"cards\", \"payments\", \"transfers\", \"topups\", \"fees\", \"security\", \"account\"]\n",
    "MAP = {\n",
    " \"cards\": \"card_arrival card_linking card_delivery_estimate card_not_working contactless_not_working getting_virtual_card card_acceptance get_physical_card visa_or_mastercard disposable_card_limits virtual_card_not_working getting_spare_card order_physical_card get_disposable_virtual_card activate_my_card card_about_to_expire apple_pay_or_google_pay card_swallowed supported_cards_and_currencies pin_blocked change_pin\",\n",
    " \"payments\": \"pending_card_payment declined_card_payment reverted_card_payment? request_refund Refund_not_showing_up transaction_charged_twice pending_cash_withdrawal wrong_amount_of_cash_received atm_support declined_cash_withdrawal\",\n",
    " \"transfers\": \"cancel_transfer transfer_not_received_by_recipient declined_transfer pending_transfer transfer_timing beneficiary_not_allowed failed_transfer transfer_into_account receiving_money balance_not_updated_after_bank_transfer\",\n",
    " \"topups\": \"automatic_top_up pending_top_up top_up_limits top_up_reverted topping_up_by_card verify_top_up top_up_by_cash_or_cheque top_up_failed balance_not_updated_after_cheque_or_cash_deposit\",\n",
    " \"fees\": \"exchange_rate card_payment_wrong_exchange_rate extra_charge_on_statement fiat_currency_support exchange_via_app card_payment_fee_charged wrong_exchange_rate_for_cash_withdrawal exchange_charge cash_withdrawal_charge transfer_fee_charged top_up_by_bank_transfer_charge top_up_by_card_charge\",\n",
    " \"security\": \"lost_or_stolen_card compromised_card card_payment_not_recognised direct_debit_payment_not_recognised cash_withdrawal_not_recognised lost_or_stolen_phone\",\n",
    " \"account\": \"age_limit edit_personal_details why_verify_identity unable_to_verify_identity verify_my_identity verify_source_of_funds terminate_account passcode_forgotten country_support\",\n",
    "}\n",
    "INTENT2TEAM = {intent: team for team, intents in MAP.items() for intent in intents.split()}\n",
    "df[\"team\"] = df[\"category\"].map(INTENT2TEAM)\n",
    "assert df[\"team\"].notna().all()\n",
    "df[\"team\"].value_counts()"
   ],
   "id": "cell-9"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Load the open model\n",
    "\n",
    "We use the best available device: CUDA, Apple's MPS, or the CPU."
   ],
   "id": "cell-10"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
    "dtype = torch.float16 if device != \"cpu\" else torch.float32\n",
    "tok = AutoTokenizer.from_pretrained(MODEL_ID)\n",
    "model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=dtype).to(device).eval()\n",
    "print(MODEL_ID, \"on\", device, \"| vocabulary:\", len(tok), \"tokens\")"
   ],
   "id": "cell-11"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Find the first token of every option\n",
    "\n",
    "The model scores *tokens*, not words, so we need the token each team name starts with. We include the capitalised and space-prefixed spellings, because the model may start its answer with any of them. Each team must own its own tokens, or two options would share probability."
   ],
   "id": "cell-12"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FIRST_IDS = {}\n",
    "for team in TEAMS:\n",
    "    ids = {tok.encode(form, add_special_tokens=False)[0]\n",
    "           for form in (team, team.capitalize(), \" \" + team, \" \" + team.capitalize())}\n",
    "    FIRST_IDS[team] = sorted(ids)\n",
    "owners = [i for ids in FIRST_IDS.values() for i in ids]\n",
    "assert len(owners) == len(set(owners)), \"two teams share a first token; rename an option\"\n",
    "{team: [tok.decode([i]) for i in ids] for team, ids in FIRST_IDS.items()}"
   ],
   "id": "cell-13"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. The one-step decision\n",
    "\n",
    "The prompt asks for the team name only, and thinking is switched off so the very next token is the answer. We run **one forward pass**, take the log-probabilities at the last position, and add up each team's first-token probabilities. We return the team scores (as log-probabilities), the normalised probabilities, and how much of the model's total probability the seven options held."
   ],
   "id": "cell-14"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math, time\n",
    "\n",
    "SYSTEM = (\"You route messages from a bank's customers to the team that should handle them. \"\n",
    "          \"Teams: \" + \", \".join(TEAMS) + \". Answer with the team name only.\")\n",
    "\n",
    "def build_prompt(text, think=False):\n",
    "    messages = [{\"role\": \"system\", \"content\": SYSTEM}, {\"role\": \"user\", \"content\": text}]\n",
    "    return tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False,\n",
    "                                   enable_thinking=think)\n",
    "\n",
    "@torch.no_grad()\n",
    "def decide(text):\n",
    "    ids = tok(build_prompt(text), return_tensors=\"pt\").to(device)\n",
    "    logp = torch.log_softmax(model(**ids).logits[0, -1].float(), dim=-1)\n",
    "    scores = {t: torch.logsumexp(logp[FIRST_IDS[t]], dim=0).item() for t in TEAMS}\n",
    "    mass = sum(math.exp(s) for s in scores.values())\n",
    "    probs = {t: math.exp(s) / mass for t, s in scores.items()}\n",
    "    return scores, probs, mass"
   ],
   "id": "cell-15"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage Example\n",
    "\n",
    "### Try it on one message\n",
    "\n",
    "This is the message the film follows. Look at the bars *before* the model has written anything."
   ],
   "id": "cell-16"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Is there an option to top up a with cheque?\"\n",
    "start = time.perf_counter()\n",
    "scores, probs, mass = decide(text)\n",
    "print(f\"{(time.perf_counter() - start) * 1000:.0f} ms | the seven options held {mass:.4f} of all next-token probability\")\n",
    "for team, p in sorted(probs.items(), key=lambda kv: -kv[1]):\n",
    "    print(f\"{team:>10} {p:8.4f} {'#' * int(p * 40)}\")"
   ],
   "id": "cell-17"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Route a batch\n",
    "\n",
    "Now the same thing on every message. We keep the scores so we can recalibrate later without running the model again."
   ],
   "id": "cell-18"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = []\n",
    "decide(\"warm up\")\n",
    "for text, truth in zip(df[\"text\"], df[\"team\"]):\n",
    "    start = time.perf_counter()\n",
    "    scores, probs, mass = decide(text)\n",
    "    pred = max(probs, key=probs.get)\n",
    "    rows.append({\"text\": text, \"truth\": truth, \"pred\": pred, \"conf\": probs[pred],\n",
    "                 \"ms\": (time.perf_counter() - start) * 1000, \"mass\": mass,\n",
    "                 **{f\"s_{t}\": scores[t] for t in TEAMS}})\n",
    "res = pd.DataFrame(rows)\n",
    "print(f\"accuracy {(res.pred == res.truth).mean():.1%} | median {res.ms.median():.0f} ms per message\")\n",
    "res[[\"text\", \"truth\", \"pred\", \"conf\"]].head(10)"
   ],
   "id": "cell-19"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison\n",
    "\n",
    "### The chatbot way: let it write, then parse\n",
    "\n",
    "The same model, with thinking on, writes out its reasoning and then its answer. We time it and check whether the written answer matches the one-step pick."
   ],
   "id": "cell-20"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "@torch.no_grad()\n",
    "def write_answer(text, max_new_tokens=1024):\n",
    "    ids = tok(build_prompt(text, think=True), return_tensors=\"pt\").to(device)\n",
    "    start = time.perf_counter()\n",
    "    out = model.generate(**ids, max_new_tokens=max_new_tokens, do_sample=False)\n",
    "    seconds = time.perf_counter() - start\n",
    "    new = out[0, ids[\"input_ids\"].shape[1]:]\n",
    "    reply = tok.decode(new, skip_special_tokens=True)\n",
    "    final = re.sub(r\"(?s)<think>.*?</think>\", \"\", reply).lower()\n",
    "    said = next((t for t in TEAMS if t in final.replace(\"-\", \"\").replace(\" \", \"\")), None)\n",
    "    return said, seconds, len(new), reply\n",
    "\n",
    "written = []\n",
    "for _, r in res.head(N_WRITTEN).iterrows():\n",
    "    said, seconds, n_tokens, reply = write_answer(r.text)\n",
    "    written.append({\"text\": r.text, \"truth\": r.truth, \"one_step\": r.pred, \"written\": said,\n",
    "                    \"seconds\": seconds, \"tokens\": n_tokens})\n",
    "    print(f\"{seconds:5.1f}s {n_tokens:4d} tokens | one step: {r.pred:>9} | written: {str(said):>9} | truth: {r.truth}\")\n",
    "w = pd.DataFrame(written)\n",
    "print(f\"\\none step: {res.ms.head(N_WRITTEN).median() / 1000:.2f}s median | written: {w.seconds.median():.1f}s median, \"\n",
    "      f\"{w.tokens.median():.0f} tokens | agreement {(w.one_step == w.written).mean():.0%}\")"
   ],
   "id": "cell-21"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is its confidence honest? The calibration check\n",
    "\n",
    "Put the answers in groups by how sure the model was, and count how often each group was right. For a calibrated model, the two columns match."
   ],
   "id": "cell-22"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "BINS = [0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99, 1.0001]\n",
    "\n",
    "def reliability(conf, correct):\n",
    "    conf, correct = np.asarray(conf), np.asarray(correct, dtype=float)\n",
    "    table, ece = [], 0.0\n",
    "    for lo, hi in zip(BINS[:-1], BINS[1:]):\n",
    "        m = (conf >= lo) & (conf < hi)\n",
    "        if m.any():\n",
    "            table.append({\"confidence\": f\"{lo:.2f}-{min(hi, 1):.2f}\", \"n\": int(m.sum()),\n",
    "                          \"mean confidence\": conf[m].mean(), \"accuracy\": correct[m].mean()})\n",
    "            ece += m.sum() * abs(conf[m].mean() - correct[m].mean())\n",
    "    return pd.DataFrame(table), ece / len(conf)\n",
    "\n",
    "table, ece = reliability(res.conf, res.pred == res.truth)\n",
    "print(f\"Expected Calibration Error: {ece:.3f}  (0 is perfectly honest)\")\n",
    "table.round(3)"
   ],
   "id": "cell-23"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at the most confident mistakes\n",
    "\n",
    "These are the dangerous ones. Software acts on a tall bar."
   ],
   "id": "cell-24"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong = res[res.pred != res.truth].sort_values(\"conf\", ascending=False)\n",
    "wrong[[\"text\", \"truth\", \"pred\", \"conf\"]].head(10)"
   ],
   "id": "cell-25"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make the number honest: temperature scaling\n",
    "\n",
    "We split the labelled messages in half. On the first half we find the one temperature `T` that makes the probabilities most honest (lowest negative log-likelihood). Then we measure the second half, which the fit never saw. `T > 1` means the model was overconfident."
   ],
   "id": "cell-26"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "S = res[[f\"s_{t}\" for t in TEAMS]].to_numpy()\n",
    "y = np.array([TEAMS.index(t) for t in res.truth])\n",
    "half = len(res) // 2\n",
    "\n",
    "def softmax(z):\n",
    "    z = z - z.max(axis=1, keepdims=True)\n",
    "    e = np.exp(z)\n",
    "    return e / e.sum(axis=1, keepdims=True)\n",
    "\n",
    "def nll(T, s, labels):\n",
    "    p = softmax(s / T)\n",
    "    return -np.log(p[np.arange(len(labels)), labels] + 1e-12).mean()\n",
    "\n",
    "temps = np.exp(np.linspace(np.log(0.5), np.log(50), 400))\n",
    "T = temps[np.argmin([nll(t, S[:half], y[:half]) for t in temps])]\n",
    "\n",
    "p_test = softmax(S[half:] / T)\n",
    "conf_after = p_test.max(axis=1)\n",
    "pred_after = p_test.argmax(axis=1)\n",
    "correct_test = pred_after == y[half:]\n",
    "_, ece_before = reliability(res.conf[half:], (res.pred == res.truth)[half:])\n",
    "table_after, ece_after = reliability(conf_after, correct_test)\n",
    "print(f\"T = {T:.2f} | ECE on unseen half: before {ece_before:.3f} -> after {ece_after:.3f}\")\n",
    "table_after.round(3)"
   ],
   "id": "cell-27"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot it: the reliability diagram\n",
    "\n",
    "A calibrated model sits on the diagonal. Points below it are overconfident."
   ],
   "id": "cell-28"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "_, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.plot([0, 1], [0, 1], \"--\", color=\"grey\", label=\"honest\")\n",
    "before, _ = reliability(res.conf[half:], (res.pred == res.truth)[half:])\n",
    "ax.plot(before[\"mean confidence\"], before[\"accuracy\"], \"o-\", label=\"raw model\")\n",
    "ax.plot(table_after[\"mean confidence\"], table_after[\"accuracy\"], \"s-\", label=f\"after T = {T:.1f}\")\n",
    "ax.set_xlabel(\"how sure the model said it was\")\n",
    "ax.set_ylabel(\"how often it was right\")\n",
    "ax.set_xlim(0, 1.02); ax.set_ylim(0, 1.02); ax.legend()\n",
    "plt.show()"
   ],
   "id": "cell-29"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The confidence gate\n",
    "\n",
    "Pick the lowest threshold whose routed messages reach the target accuracy on the calibration half. The target is a business choice: how often can an automatic route be wrong before a person should have looked? Then check it on the unseen half: how many messages route automatically, and how often those are right."
   ],
   "id": "cell-30"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TARGET = 0.70   # a business choice; the small default model tops out near here, with Qwen3-8B try 0.90\n",
    "\n",
    "p_cal = softmax(S[:half] / T)\n",
    "conf_cal, correct_cal = p_cal.max(axis=1), p_cal.argmax(axis=1) == y[:half]\n",
    "\n",
    "# the trade-off on the calibration half: a higher gate routes fewer messages, more of them right\n",
    "trade = []\n",
    "for th in np.round(np.arange(0.30, 0.96, 0.05), 2):\n",
    "    m = conf_cal >= th\n",
    "    if m.sum() >= 10:\n",
    "        trade.append({\"gate\": th, \"routed automatically\": m.mean(), \"accuracy of routed\": correct_cal[m].mean()})\n",
    "trade = pd.DataFrame(trade)\n",
    "print(trade.round(3).to_string(index=False))\n",
    "\n",
    "ok = trade[trade[\"accuracy of routed\"] >= TARGET]\n",
    "if ok.empty:\n",
    "    print(f\"\\nNo gate reaches {TARGET:.0%} on this model: send everything to a person, or use a bigger model.\")\n",
    "else:\n",
    "    gate = ok[\"gate\"].min()\n",
    "    m = conf_after >= gate\n",
    "    print(f\"\\ngate {gate:.2f} | unseen half: {m.mean():.0%} routed automatically at {correct_test[m].mean():.1%} accuracy, \"\n",
    "          f\"{(~m).mean():.0%} sent to a person (every message routed: {correct_test.mean():.1%})\")"
   ],
   "id": "cell-31"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Considerations\n",
    "\n",
    "- **Your mapping is part of the result.** Messages that fit two teams cap the accuracy. A well-calibrated model shows this as a *split* probability, which is exactly the signal the gate uses.\n",
    "- **Option names matter.** Give each option a distinct first token (the assertion in step 5 checks this). For long option lists, use letters or short codes in the prompt.\n",
    "- **Temperature scaling fixes the scale, not the ranking.** It makes the confidence honest but does not change which team is on top. For better picks use a bigger model, a clearer prompt, or a few labelled examples in the prompt.\n",
    "- **Re-check after every change.** A new model, prompt or kind of traffic changes the calibration. Keep a few hundred labelled messages and re-run this check.\n",
    "- **Hosted APIs:** any API that returns token log-probabilities (for example `logprobs` with `top_logprobs`) supports the same one-step read. Purpose-built decision models such as TypeSafe's Jev return a choice and a probability directly and say they are trained for calibrated confidence. Either way, the test in this notebook is how you verify that claim on your own data.\n",
    "\n",
    "## References\n",
    "\n",
    "- banking77: Casanueva et al., *Efficient Intent Detection with Dual Sentence Encoders* (2020), [dataset](https://github.com/PolyAI-LDN/task-specific-datasets), CC-BY-4.0.\n",
    "- Guo et al., *On Calibration of Modern Neural Networks* (ICML 2017): temperature scaling and ECE.\n",
    "- Kahneman, *Thinking, Fast and Slow* (2011): the System 1 / System 2 idea behind the name \"System One model\".\n",
    "- TypeSafe AI, [Introducing System One Models & Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev).\n",
    "- Qwen team, [Qwen3](https://huggingface.co/Qwen/Qwen3-1.7B) open-weight models."
   ],
   "id": "cell-32"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "📖 **Want the decisions behind agents like this one, explained without code?** [*AI Agents Made Simple*](https://europe-west1-rag-techniques-views-tracker.cloudfunctions.net/rag-techniques-tracker?notebook=all-agents-tutorials--one-step-decision-router&click=book-nb-agents&target=https%3A%2F%2Fwww.amazon.com%2Fdp%2FB0HNXRSRH2%3Fmaas%3Dmaas_adg_api_580351185840460851_static_9_573%26ref_%3Daa_maas%26tag%3Dmaas%26aa_campaignid%3Dagents-en%26aa_adgroupid%3Dgithub-notebook%26aa_creativeid%3Dgenai-decision-router&retarget=0&text=book-nb-agents) by the author of this repository has a full chapter on whether you need an agent at all: the four rungs from a single call to a router to an agent, the three questions that pick the rung, and what goes wrong when routes overlap. Available on Amazon."
   ],
   "id": "f9e03e40"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://europe-west1-genai-agents-views-tracker.cloudfunctions.net/genai-agents-tracker?notebook=all-agents-tutorials--one-step-decision-router)"
   ],
   "id": "da995209"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}