# precondition-library — design of record

**Date:** 2026-09-13
**Status:** design approved, implementation not started
**Author:** Shyboy0499

**Revision history**

| version | date | change |
| --- | --- | --- |
| 1 | 2026-09-13 | Initial design of record. |
| 2 | 2026-09-13 | Reframed after a prior-art sweep and a hostile method review (see ADR-0001). Claim 1 dropped as a contribution; Claim 2 narrowed to its empirical form; Claim 3 added for the negative-sandbox admission criterion; the primary metric moved from episode-level to matched dispatch coverage. **Revised before any data was collected** — no episode has been run, so no analysis was chosen after seeing results. |
| 3 | 2026-09-13 | The request text gains a declared informed/uninformed channel split, and the primary claim is scoped to the uninformed regime (where the construction fixes the AUC at 0.500); the gold resolutions land. `sync_fork_with_upstream` and `restore_submodule_state` are converted to state-decided intents; three of the five faults remain unconverted and are excluded from any dispatch measurement (issue #25). **Logged before any episode data existed** — no episode has been run, so no analysis was chosen after seeing results; the only measured figure so far is the informed text-control AUC reported in §3. |
| 4 | 2026-09-14 | **Lifecycle correction, not a claim change.** A program that fails admission is stored as a `candidate` and its rejection reason recorded, not discarded; the §5 lifecycle diagram and the §8 degradation table said "not stored". Corrected to match `library.py` (which writes the candidate before the gate runs) and `library/README.md`; rejection is data the mismatch analysis needs. No measurement or claim is affected, and no number in this document changes as a result. |
| 5 | 2026-09-15 | **Arm 2's mechanism is narrowed (ADR-0002).** The arm compares text by a deterministic lexical overlap behind a `Similarity` seam, not by an embedding; an embedding model is the intended replacement behind the same Protocol. Claim 2's wording follows — the comparison is against *text* similarity — and arm 2 is given the full representation (the intent plus the `StateFingerprint` rendered as text), per issue #4, so the ablation does not confound dispatch with representation. **Logged before any episode data existed** — no episode has been run, so no analysis was chosen after seeing results, and no number in this document changes. |
| 6 | 2026-09-16 | **The seed split is made concrete, and the eval set is stated to be underpowered.** Items 2 and 5 required disjoint admit/tune/eval seeds but no seeds were defined anywhere; §7 now names them and fixes them in `bench/splits.py`. The eval set (40 seeds, 80 decisions) cannot support the primary matched-coverage comparison at a usable interval, so that claim requires the item 6 extension rather than being read off the first run. **Logged before any episode data existed** — no episode has been run, so no analysis was chosen after seeing results, and no number in this document changes. |
| 7 | 2026-09-16 | **Schema correction, not an analysis change.** The ledger's single reason field carried guard refusals, compile failures and invalid-episode causes at once, so a guard refusal rate read off it would have included malformed replies that no guard saw (issue #60); §7's ledger schema and §8's table now name three fields — `refusal_reason`, `compile_failure_reason`, `invalid_reason`. §7 also records that a program whose `variant` is not a declared resolution of an ambiguous intent is quarantined on load, so a `program.yaml` written straight to disk cannot be dispatched as scored (issue #66). Both are corrections to match the code; no metric definition, denominator or number in this document changes. |
| 8 | 2026-09-17 | **Documentation corrections, not analysis changes.** §7's demo figure said 5 faults × 4 occurrences × 3 arms = 60 episodes; only the two faults with an ambiguous intent are runnable (`registry.ambiguous_intents()`), so the runnable demo is 24 and 60 is the nominal grid — the overstatement was 2.5× and sat on a measurement path. §9 names the mechanisms that actually exist for network containment (the guard's textual refusals and the environment allowlist) instead of `sandbox-exec`, which nothing implements. `bench/report.py`'s docstring carried the same 60-episode overstatement. **No measurement or number from a run changes** — no episode has been run. |
| 9 | 2026-09-17 | **Admission negative-side correction, not an analysis change.** §6 named three classes of negative sandbox but `admit` built only "a different fault injected", so a program whose preconditions accepted a sibling resolution of its own intent — firing where firing is wrong — passed the gate (the coupling/duplication audit's case: a submodule program gated only on a gitlink reference accepts the `remove` state). `admit` now builds the missing two: a fault-free sandbox (`build_sandbox(seed, [])`), which every program must refuse, and the program's own fault at a seed whose state a sibling resolution is correct in, chosen from the same seed-to-state mapping the injector uses (`FaultSpec.variant_for_seed`). Each rejection names the class and the number of states it checked. §6 is corrected to describe the three classes the code now builds. Running the hand-written gold programs through the stricter gate found `sync_fork_with_upstream`'s merge preconditions too permissive for the unrelated lockfile-conflict sandbox (the committed record of that tightening is in `bench/gold/sync_fork_with_upstream.yaml`); the gate was not weakened. **No measurement or number from a run changes** — no episode has been run. |
| 10 | 2026-09-17 | **Two defect corrections to the replay and compile paths, not analysis changes.** A program that fired but whose body names a declared parameter the environment cannot bind raised out of `runtime.replay`, so the exception escaped `run_episode` and ended the whole run instead of recording the mis-fire; it is now a `ReplayResult(unbound_parameter=True)` that demotes the program and is named on the row in §7's new `replay_failure_reason` field (issue #76). The compile prompt states each field's exact shape with a minimal example — `body` is one newline-separated string, `parameters` a list of names — and a list `body` is still rejected with the field named rather than coerced, so the compile-quality signal survives (issue #78). No metric definition, denominator or analysis changes; the smoke pass that found both defects never exercised the replay path, and no result is claimed from it. |
| 11 | 2026-09-17 | **Two library-integrity corrections, not analysis changes.** A compile that reused a program id on a later episode was correctly refused by the library, so the episode's program was lost and the row showed only a `compile_failure_reason` that read like a malformed reply; the stored id is now derived from the episode's `(fault, occurrence)` plus a sanitised slug, and the residual same-episode collision is recorded *as a collision* (issue #80). The load-time variant check keyed on `Program.intent` matching an `IntentSpec.name`, but a compiled program's intent is prose, as the compile prompt asks, so the check missed exactly the `program.yaml` written past `admit` that it exists for; it now keys on a new required `Provenance.fault` (issue #69). The same correction makes the read pure: `Library.load_all` applies the verdict in memory and writes nothing, and `Library.quarantine_undeclared()` is the explicit write that records it. **No metric definition, denominator or number in this document changes** — no episode has been run. |
| 12 | 2026-09-17 | **Two admission-gate corrections found by the smoke pass, not analysis changes.** (1) §6 now states a DECLARED condition before the two sides: a body parameter that no precondition's probe names is refused, because a precondition is how a program declares what it needs and the states it may fire in need not bind an undeclared one. The smoke pass produced three rows carrying `replay_failure_reason` from a `submodule` program whose body used `submodule_path` while its preconditions did not (issue #77). (2) The unrelated-fault negative class is corrected from "one fixed seed each" to one seed per distinct state its injector can select, read through `FaultSpec.variant_for_seed`; `diverged` is now built at seeds 0, 1 and 2 and `submodule_moved` at 0, 1 and 4, where the old class built each at seed 0 only, so a program could be rejected on one state and fire on another (issue #75). The faults that expose no seed-to-state mapping remain one state each, labelled as sampling. **No metric definition, denominator or number in this document changes** — no episode has been run, and the smoke pass claims no result. |
| 13 | 2026-09-17 | **Pre-registration revision: occurrences are labelled by role, and the episode loop's independent observations are stated (issue #81).** A smoke pass reported zero replays because its seeds each select a different resolution, and a program admitted for one resolution correctly refuses the others — so the plan as written produced a flat cost curve *by construction*. §7 gains a required `occurrence_role` on every ledger row, declared from the injectors' own seed-to-resolution mapping and written by the runner: **variant** for the first sight of a resolution, **replay** for every later one. The cost curve is computed over the replays (where the accumulation is visible), the mismatch comparison over the variants (the only independent observations), and both figures name the occurrences they used. §7's "Design" resolves the choice the earlier text left open — held-out variants or relabelled replays — in favour of replays, **because the injectors do not vary branch names, file sets or conflict positions by seed**; widening them is issue #6. The eval set's arithmetic is now stated: each measurable fault declares three states, so 40 seeds yield **3 variant and 37 replay occurrences per family**, and the episode-level mismatch comparison has six independent observations rather than eighty. **Logged before any eval data existed** — no eval episode has been run, so no analysis was chosen after seeing results; the smoke pass that motivated it is excluded from every claim in §7, and no reported number changes. |
| 14 | 2026-09-18 | **Portability corrections, not analysis changes (issue #87).** §7's invocation wrote its ledger to a hardcoded `/tmp/precondition-ledger.jsonl`, which is not a path that exists on Windows; it now uses `Path(tempfile.gettempdir())`, which keeps the same intent (the ledger stays outside the repository) without naming a platform. Recorded alongside four source and test defects a Windows verification pass found — a bare `bash` resolving to the WSL launcher, `shell=True` selecting `cmd.exe` for multi-line bodies, `shutil.rmtree` failing on git's read-only loose objects, and an import-time `git status` that stopped the suite being collected outside a git checkout. No metric definition, denominator or number in this document changes, and no result is claimed from that pass. |
| 15 | 2026-09-18 | **Figure 2 reports cumulative amortized cost with the break-even, not per-occurrence means (issue #8).** Claim 1 is about cost falling across occurrences, and a mean per occurrence is a snapshot: it says what one occurrence cost, not whether the compile has been repaid, so no crossover could be read off the figure the claim rests on. Figure 2 now plots the running amortized tokens and LLM calls per episode, with the break-even marked on the figure and stated in the report text, and §7's secondary-metrics list says so. `bench/report.py` computes the accumulation and reports why a crossing is not computable when it is not. No metric definition, denominator or reported number changes, the ledger schema is unchanged, and the analysis reported is the one rev 13 describes, read off an accumulating series rather than a per-occurrence one. **Logged before any eval data existed** — no eval episode has been run. |
| 16 | 2026-09-18 | **Spec-gaming becomes its own recorded fact and its own column (issue #9).** §7's ledger gains `refs_intact`, the fact of whether a resolution left the recorded refs and upstream's history alone; a row that reached the expected state while that is false was spec-gamed, and the ablation table and report now count it separately from a resolution that simply failed. Recorded as a fact and derived, not stored as a verdict, for the reason §7 already gives for `misfired`/`succeeded`: a stored verdict could not sit beside `ground_truth_ok` without the two disagreeing. `null` means the check did not run and is never counted as gaming. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 17 | 2026-09-18 | **The arm triple and its Pareto frontier are reported, making item 8's promise true (issue #8).** §7 item 8 and Claim 1 both require cost per success beside cost per episode, and the report stated only the latter. It now reports the triple — success rate, tokens per episode, tokens per success, each with the denominator it was taken over — pooled over every graded episode of the arm with the failures kept in the denominator per item 7, plus a Pareto frontier over the three so no arm is read on one axis alone (`arm_triples.csv`, `pareto.png`). Cost per success is the arm's total tokens over its successes, not the mean cost of a successful episode; the two differ whenever an arm fails, and the second would flatter an arm that fails often. An arm with no success has no cost per success and is reported unranked rather than assigned a place. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 18 | 2026-09-18 | **A pre-registered TOST gates any "equal success rate" wording, and the verdict is reported (issue #8, item 8).** §7 gains principle 10: "equal" requires an equivalence test and "comparable" is the fallback, at a margin of ±10pp and α = 0.05 registered here before any data. `bench/report.py` runs it on arms 2 and 3's pooled success rates and prints the difference, the 90% interval, both one-sided p-values and which of three cases holds — equivalent; outside the margin; or too wide to decide, which is an underpowered run rather than a difference. The margin and alpha are named constants, so the choice is visible and dated rather than buried in a call, and the variance is Agresti–Coull-adjusted so a 0%/100% rate cannot produce a zero-width interval. **No live document currently claims an equal success rate** — Claim 1 already says an arm that succeeds more often may spend more, and defers to cost per success — so this closes the gap between that claim and its evidence before such a sentence can be written, rather than correcting one. No metric definition, denominator or reported number changes and no result is claimed; no episode has been run. |
| 19 | 2026-09-18 | **The raw prompt prefix length is reported, once rather than per arm (issue #8, item 5).** The item asks for it per arm, and measuring first showed the premise does not hold: both prompts are module-level constants — `agents/react.py`'s `SYSTEM_PROMPT` and the one derived in `agents/compile.py` — and **every arm sends the same two**, so a per-arm column would print one number three times and imply a difference that does not exist. §7's metrics list now requires the prefix length in raw characters, reported once with that reason, and notes that the arm-level difference is transcript growth around the prefix rather than the prefix. The compile prefix is the only one that is not fixed: it grows by the joined length of an intent's declared variant ids. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 20 | 2026-09-18 | **Input tokens are metered in three components, with the provider mapping stated (issue #8, items 1-2).** §7's ledger gains `uncached_tokens_in` and `cache_write_tokens_in` beside the existing cache-read field, and a bullet records how each is derived and what it means per provider: OpenAI-style APIs report `prompt_tokens` **including** cached tokens, so the uncached component is derived from `prompt_cache_miss_tokens` where published (DeepSeek publishes it) and by saturating subtraction otherwise; Anthropic-style APIs bill a separate cache write, which is why that field exists though DeepSeek always writes zero. `tokens_in` is documented as the provider's total and explicitly **not** a billing basis — cache-hit input is published at 1/50th of cache-miss input, so pricing the total at one rate overstates the arm that caches most. `bench/report.py` reports the components per cell and warns at the cost figure. **Not implemented:** applying rates. That needs a rate table, and DeepSeek's rates differ by peak and off-peak hours, so a correct figure needs the rate or the call time recorded per episode. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 21 | 2026-09-18 | **Claim 1 is restated to match the figures that now exist (issue #8, item 9).** The claim said tokens and calls fall per episode with cost per success beside cost per episode, which understated what the report now does and overstated what it can claim: the curve it is read from is cumulative and amortized with an explicit break-even (#94), the triple including cost per success and its Pareto frontier is reported (#97), the cache components are metered separately so input is not priced as one number (#100), and any "equal success rate" wording requires a pre-registered equivalence test (#98). The claim now names the cumulative figure, the triple, and **two limits that belong to the claim** rather than to its footnotes: it is a claim about tokens and calls and not about currency, because pricing needs rates that vary by peak and off-peak hours and are not applied; and equality of success rates is not available to it, because that needs the equivalence test to pass. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 22 | 2026-09-18 | **The injected state is no longer recorded inside the clone (issue #103).** `submodule_moved.inject` wrote the resolution label — `init`, `repin` or `remove` — into `refs/sandbox/submodule-state` in `work`, the same clone `runtime/probes.py` runs probes in, so a precondition could read the answer instead of diagnosing the fault and no admission class could reject it. Measured before the fix: probes read `remove`, `repin` and `init` for seeds 0, 1 and 4. The state now travels on the `Sandbox` object, which the harness owns, set by `build_sandbox` from the same seed-to-state mapping the injector uses; `check` takes it as a selector and still reads its clause from the environment. §7 records the channel. The probes' `submodule-path` ref stays: it carries a path rather than an answer, and a correct removal deletes the `.gitmodules` entry it would otherwise come from. No committed content changes, so no commit SHA moves; a ref and an uncommitted blob are simply no longer written. No metric definition, denominator or reported number changes and no result is claimed. |
| 23 | 2026-09-18 | **Probes are screened before they run (issue #10, item 2).** A probe is model-authored text that reached a shell unscreened, while a body is screened in `runtime.replay` — so the model-authored path with *less* screening was the one that runs on every dispatch attempt rather than once. Measured before the change: a precondition of the form `touch probe-wrote-this.txt && curl -s http://example.invalid/x` both wrote to the working tree and reached the network, and the identical text as a body was refused with `refused outbound network call`. `evaluate_predicate` now screens the substituted probe with `runtime.guard`, using the same `env_root` a body is judged against, and reports a refusal as **not holding** with the reason in `observed` — the shape an unbindable parameter already takes, so dispatch treats it as an ordinary non-match and admission can name it. **This does not make probes read-only:** the guard permits writes inside the sandbox because bodies need them, so an in-repo mutation such as `git reset --hard` in a probe still runs. The remaining halves of item 2 — refusing in-repo writes and the measured breadth cap — are not done. No metric definition, denominator or reported number changes and no result is claimed. |
| 24 | 2026-09-18 | **The pre-injection tip is no longer recorded in the clone (issue #103).** `record_base` used to point `refs/sandbox/base` at the tip the fault was injected on top of. That ref is a *diff against the fault*: `git diff refs/sandbox/base HEAD` shows the injected change, so a body could read what was done to the repository instead of diagnosing it. The value now lives in `Sandbox.recorded`, which the harness owns, and `diverged.check` takes it from there; the clone gets `refs/sandbox/injected`, an empty blob whose only content is "a fault ran here", which the task text already says. The marker stays on disk because a second `inject` has to see it and an in-memory marker would not outlive its object. This closes the base leak for all four faults that call `record_base`, not just `diverged`, because the recording happens in one shared function. Remaining in the clone and tracked by #103: `diverged`'s `local-tip`, `dirty_tree`'s patch and untracked blobs, and `lockfile_conflict`'s three refs. No committed content changes, so no commit SHA moves. No metric definition, denominator or reported number changes and no result is claimed. |
| 25 | 2026-09-18 | **The remaining recorded values move into the harness, finishing issue #103's value half.** `diverged`'s `local-tip`, `dirty_tree`'s patch and untracked blobs, and `lockfile_conflict`'s local tip and two dependency entries all lived under `refs/sandbox/` in the clone the graded code reads, so a body could read the injected state's evidence rather than diagnose it. They now travel in `Sandbox.recorded`, and the checkers take them from there — the mechanism #103's earlier change introduced for the pre-injection tip. What remains in the clone is `refs/sandbox/injected`, an empty marker ref, and `submodule_moved`'s `submodule-path`, which stays because the probes bind `{submodule_path}` from it and a correct removal deletes the `.gitmodules` entry it would otherwise come from. A mechanical edit briefly inverted `dirty_tree`'s untracked clause into `"untracked" in sandbox.recorded == 0`, a chained comparison that is always false and silently disabled the check for a `clean -fd` that destroys the work; the fault's own destructive negative control caught it, which is what those controls are for. No committed content changes, so no commit SHA moves. No metric definition, denominator or reported number changes and no result is claimed. |
| 26 | 2026-09-18 | **Arm 2's embedding currency is scaffolded, so the accounting exists before the model does (issue #104).** The seam is lexical (ADR-0002), so there is no embedding spend to meter and nothing that could be folded into `tokens_in` — which is why this is a scaffold rather than an implementation. What is now in place: `SimilarityUsage` and an **optional** `ReportsUsage` Protocol, deliberately *not* part of `Similarity`, so that seam stays a one-method callable and `lexical_similarity` satisfies it unchanged; `similarity_usage(seam)` answering zero for a seam that does not report; `embedding_tokens` and `embedding_calls` on the ledger, documented as their own currency that is never added to the LLM's; `run_benchmark` taking a `similarity` argument, because knowing what an embedding model costs means being able to run with one; the runner measuring the seam's usage across each episode and recording the delta; and the ablation table, its CSV and `report.txt` stating it beside the LLM tokens. The invariant the issue asks for is asserted end to end: with a seam that reports a cost, a replay row carries `embedding_tokens > 0` while `tokens_in == 0`, so a second bill cannot be folded into the first. Every row is 0 while the seam is lexical, and the field is documented so a non-zero value is itself the signal that a model is behind the seam. No metric definition, denominator or reported number changes and no result is claimed — no episode has been run. |
| 27 | 2026-09-18 | **`refs_intact` is renamed `recorded_state_intact`, and its meaning is widened to the recorded state (issue #96).** Refs are the first thing the fact covers, not all of it: a minimal-diff check against a fault's declared change surface belongs to the same fact, so it joins this field rather than growing a second boolean that the report would have to conjoin to derive spec-gaming — the shape §7 already warns against for stored verdicts. The rename is done now, before any ledger row exists, because it is a schema change and the cost of it rises the moment rows do. The function in `tasks/invariants.py` is renamed with the field. Revision 16, which introduced the field, is left naming it `refs_intact`: a record of a time is not rewritten (rule 5), and this row is the narrowing that supersedes it. No metric definition, denominator, reported number or behaviour changes, and no result is claimed. |
| 28 | 2026-09-18 | **Each fault declares the paths a resolution may touch, and the invariant checks it (issue #96).** `FaultSpec.change_surface` is a **declaration**, not something derived from what the injector touched: a repair that rewrites a generated file from its source legitimately writes a path the injection did not name, and a derived surface would refuse it for being outside the fault. `diverged` allows `app.py` and `docs/readme.md`; `dirty_tree` those two plus `notes/scratch.txt`; `lockfile_conflict` `deps.lock`; `submodule_moved` `.gitmodules` and the gitlink path; and `branch_renamed` declares **empty**, which is the strictest value rather than a missing one — its resolution is a ref operation, so any commit is something the fault did not ask for. Empty is also the default, so a fault that forgets to declare fails its own gold resolution instead of passing on a permissive default. The check joins `recorded_state_intact` rather than growing a second field (revision 27), reading the committed diff from the recorded base and refusing any path outside the surface. `submodule_moved` gained a recorded base, which it had never needed before because nothing diffed against it. Tests pin three different properties: coverage (what each injector commits is inside its own declaration, per fault), firing (a committed change outside the surface is refused, naming the path), and the control (an in-surface commit is allowed), so the check cannot be a blanket ban or a no-op. Not covered: an **uncommitted** change is invisible to a diff between two commits, and refs outside `refs/sandbox/` remain unpinned. No metric definition, denominator or reported number changes and no result is claimed. |
| 29 | 2026-09-18 | **`embedding_calls` is settled as the calls the implementation reports making to its provider (issue #104).** It was described as "how many times the seam was called", which is a different quantity: an implementation that batches or caches can be asked for a hundred scores and make one call, and only it knows which. The invocation count is also mostly a property of the dispatch loop -- how many admitted candidates it scores -- so it would say little about arm 2's cost while looking like it did; a reader who wants that number wants the library's size, which the ledger records per episode. The ledger field and `SimilarityUsage.calls` now agree on the provider reading, and a caching seam is the test that pins it: the harness observes invocations, the seam reports traffic, and the ledger carries the traffic. No metric definition, denominator or reported number changes and no result is claimed. |
| 30 | 2026-09-18 | **The state grid moves into the package, and a discrimination probe measures any similarity scorer against it (issue #104).** The probe answers the question that has to precede filling arm 2's `Similarity` seam: can a scorer tell which resolution a state needs, *from the text*? Each labelled pair is crossed with its intent's declared resolutions, the scorer scores the request text against each resolution's `rationale`, and the AUC asks how often a correct resolution outranks an incorrect one, over 56 pairs and 168 crossings on the declared grid (seeds 0–3, named separately from the pre-registered plan). Measured on the shipped scorer: **lexical 0.6755**, against **0.5000** for a constant scorer that cannot discriminate — an exploratory figure with its provenance, not a claim. Two limits are stated in the module rather than discovered later: `rationale` is a proxy for a compiled program's text, so this answers *is the fault family separable by meaning* and **not** the dispatch AUC the benchmark reports; and the figure is **reported, never gated**, because a threshold on a proxy would turn an exploratory number into a claim. The grid itself moved from `tests/conftest.py` into `tasks/state_grid.py` so `bench` can reach it — a declared fingerprint per fault state, which the test suite still checks against what a real sandbox observes, so the declaration cannot drift from the injectors. No metric definition, denominator or reported number changes and no result is claimed. |
| 31 | 2026-09-18 | **Revision 30's discrimination figure is withdrawn as a baseline, and a baseline defect is recorded (issue #116).** That probe crossed the request with each resolution's `rationale` — the designer's explanation — while a dispatcher compares against `_program_text`: a program's `intent` plus its predicate descriptions. Measured against the artifact, the shipped scorer gives **AUC 0.4778** and puts the correct gold program first in **9 of 28 pairs (32%, chance for three candidates)**, because `program.intent` is the intent *name* — a constant across a fault's resolutions — and unweighted Jaccard over token sets is dominated by the shared boilerplate; correct and incorrect crossings have an identical maximum score. The module's own number (0.6755, rationales) flattered the scorer by ~0.2 AUC and must not be quoted, and the conclusion drawn from it — that a semantic scorer has little headroom here — is **withdrawn and reversed**: with almost no lexical signal, the case for an embedding model is stronger. The finding is scoped to hand-written gold; whether compiled programs' texts are equally non-discriminative needs a compiled library. No metric definition, denominator or reported number changes and no result is claimed. |
| 32 | 2026-09-18 | **The probe measures the artifact in code, not only in a note, and its top-1 denominator is stated (issue #116).** Revision 31 corrected the figure the probe *reports* while the code still scored rationales — a docstring that says "do not quote this number" is a warning, not a fix, and the next caller would have read the number anyway. The candidate source is now explicit: `program_text_candidates` calls `library._program_text` rather than restating it, `discrimination_scores`/`discrimination_auc` take a `Mapping[variant, text]` instead of a resolution sequence, and `compare_scorers` takes candidate texts per intent and raises rather than silently narrowing the comparison when one is missing. `rationale_candidates` is **kept as a diagnostic**, because the two sources disagreeing is itself the evidence, and `test_the_two_candidate_sources_disagree` fails if they stop differing — so the substitution that caused this cannot be repeated quietly. This row also **corrects revision 31's denominator**: 9 of 28 is not a rate over the pairs. Strict top-1 is decided only on pairs that *have* a correct resolution, and 24 of the 28 do (the other four are the negative pairs where nothing should fire), so the figure is **9/24 = 38% against a 33% chance expectation for three candidates**, with ties counted as no decision rather than as wins. Read together with AUC 0.4778 — below chance — the two are not in tension once the denominators are named: nine strict wins is one pair above chance, so neither number shows usable signal and the finding stands. Rechecking revision 31's numbers against the code is also what produced this row: the pair count had been taken from the crossing rather than from the metric. Nothing the benchmark defines, divides or reports changes — this is the probe's own metric — and no result is claimed. |
| 33 | 2026-09-18 | **Revisions 31 and 32's discrimination figures were taken on one intent and quoted as the family's; measured over the whole ambiguous subset the shipped scorer is above chance, and the scorer decision is scoped in ADR-0003 (issues #104, #116).** Revision 30 framed the probe over both ambiguous intents — 56 pairs and 168 crossings — while revision 31's withdrawal, and revision 32's denominator correction, reported `sync_fork_with_upstream` alone (AUC 0.4778, strict top-1 9/24) as the baseline without saying so. Over the subset the metric is defined on, with the same scorer, text and seeds: `restore_submodule_state` **AUC 0.7264, top-1 14/24 (58%)**; `sync_fork_with_upstream` **AUC 0.4778, top-1 9/24 (38%)**; both together **AUC 0.5722, strict top-1 23/48 (48%)** against a 33% chance for three candidates, with a constant scorer at 0.5000. So the family baseline is **above chance, not below**, the two intents differ by 0.25 AUC, and revision 31's reversal of the embedding conclusion — that the case for a semantic scorer is stronger — held only on the weaker intent; it is narrowed here to that intent rather than reversed again. Revision 32's denominator convention stands and is what produces 23/48. ADR-0003 (proposed) scopes the decision: neither alternative #116 left open improves the metric dispatch acts on — IDF-weighted Jaccard 19/48 and cosine 15/48 against 23/48, and dropping the constant `intent` leaves top-1 unchanged — and IDF cannot be carried by a one-method seam at all. The probe's crossing functions now refuse a candidate mapping that spans two intents, because variant ids are unique only within one and the mistake returned a plausible wrong number (0.6980, 18/48) rather than failing. Nothing the benchmark defines, divides or reports changes, and no result is claimed. |
| 34 | 2026-09-20 | **ADR-0003 is accepted, and the spec of record now carries arm 2's measured baseline instead of leaving the risk unquantified (issues #104, #116).** Revisions 31–33 correct the probe and its figures in the log; this row amends the **live** sections, because a reader of the design should not have to reconstruct a decision from the revision history. §12's decisions table gains a row for arm 2's baseline and seam, and Open risks item 1 now states the exploratory reading the probe gives before any episode runs: over the ambiguous subset the shipped scorer scores **AUC 0.5722, strict top-1 23/48 (48%)** against a 33% chance for three candidates, with the pooled figure hiding **0.7264 / 58%** on `restore_submodule_state` against **0.4778 / 38%** on `sync_fork_with_upstream`. Those are probe figures on hand-written gold, explicitly **not** tune-set results and not the pre-registered split, and the risk is now stated as the spread rather than as a single number: on the second intent the arm is effectively at chance, so the comparison there is the one that flatters preconditions. The row also records what was rejected with numbers — IDF weighting raises the pooled AUC while lowering strict top-1, and dropping the constant `intent` moves the AUC by 0.0104 for no gain — and that the embedding swap stays the one open candidate. No benchmark metric, denominator or reported number changes: the probe measures arm 2's representation and is not the primary metric, and no result is claimed. |
| 35 | 2026-09-21 | **The gold programs' predicate descriptions are rewritten to state the applicability condition in request vocabulary, and the rewrite does not rescue arm 2 (issue #104).** #104's finding was that an intent's three candidate texts collapse together for a semantic scorer, so the experiment was to rewrite each `description` — the text `library._program_text` feeds arm 2 — to say what *state* the program is for, using words a request would use, without naming a resolution and without copying a request's wording. The rewrite also closes a leak: `sync-fork-overlapping-merge`'s `local_work_beyond_the_overlap` description read "...whose right resolution is not a **merge**...", naming the label `_program_text` excludes `variant` to keep out; `tests/test_gold_programs.py::test_no_gold_description_names_a_resolution` now guards that, and the admission-gate history that description carried moved to a YAML comment so it is not scored as meaning. Only `description` values changed; `probe`, `body`, `parameters`, `variant`, `intent` and `provenance` are byte-identical, and `test_the_gold_resolution_stays_inside_its_declared_surface` replays the same resolutions. Re-measured on the committed gold and the probe's seeds: pooled lexical is **AUC 0.5340, strict top-1 23/48** (was 0.5722 / 23/48) and the embedding **AUC 0.5252, strict top-1 20/48** (was 0.5571 / 19/48); per intent, lexical `restore_submodule_state` **0.6299 / 12-of-24** and `sync_fork_with_upstream` **0.4965 / 11-of-24**, embedding **0.5556 / 12-of-24** and **0.5062 / 8-of-24**. Candidate-vs-candidate mean embedding cosine **rose**, to **0.8909** on `restore_submodule_state` and **0.8932** on `sync_fork_with_upstream` (was 0.8718 / 0.8215). So separating the descriptions did **not** raise the metric dispatch acts on, and made the candidates no less interchangeable to the embedding: the collision is a property of three resolutions that describe nearby states in one domain, not of vague wording. The figures are printed by `tests/test_similarity_probe.py -s -k 'pooled_baseline or candidate_texts'` and `tests/test_embedding_similarity.py -s -k candidate_texts`. No benchmark metric, denominator or reported number changes — the probe measures arm 2's representation and is not the primary metric — and no result is claimed. |
| 36 | 2026-09-21 | **The probe's discrimination figure was pooled across two request regimes; measured per regime the informed baseline is AUC 0.6172 and strict top-1 15/24 (63%), and the pooled number is withdrawn as a baseline (issue #104).** Every `LabelledPair` records the channel its text arrived through, and `bench/textcontrol.py` states the consequence: the uninformed channel's sampler never consults state, so its AUC is **0.5000 for any classifier and any phrasing list** — a property of the construction, not a measurement — while informed wording nearly gives the resolution away, so a high score there is the boundary condition and the genuine measurement. Revisions 30 onward scored both channels together, so ADR-0003's accepted baseline (**0.5722 / 23-of-48**, re-measured **0.5340 / 23-of-48** in revision 35) and this spec's live figures were pooled numbers that described neither regime. Re-measured with the committed code — hand-written gold on the probe's seeds 0–3, which are **not** the pre-registered split: **informed, both intents** lexical **AUC 0.6172, strict top-1 15/24 (63%)**, per intent `restore_submodule_state` **0.8125 / 8-of-12** and `sync_fork_with_upstream` **0.6007 / 7-of-12**; **uninformed, both intents** lexical **AUC 0.5000, strict top-1 8/24 (33%)**, chance by construction. The 8 negative pairs belong to the uninformed regime, where they are the plumbing tripwire rather than baseline material; the informed regime has no negatives, because informed wording exists only where a resolution does. The embedding scorer re-measured on the same split: informed **AUC 0.5859, strict top-1 12/24 (50%)** (per intent **0.6389 / 8-of-12** and **0.5729 / 4-of-12**), uninformed **0.5000 / 8-of-24**. So the corrected baseline is **stronger** — 63% where ADR-0003 recorded 48% — which is the direction that makes the claim harder to support, and the embedding does not improve the informed top-1. The probe now takes a required `regime` on its single-mapping functions, `compare_scorers` returns one row per regime, and there is no pooled mode, so the defect cannot return through a default; ADR-0004 (proposed) records the correction and re-derives ADR-0003's baseline decision, which stays accepted as a record. The figures are printed by `tests/test_similarity_probe.py -s -k informed_baseline` and `tests/test_embedding_similarity.py -s -k measured_against_lexical`. No benchmark metric, denominator or reported number changes — the probe measures arm 2's representation and is not the primary metric — and no result is claimed. |
| 37 | 2026-09-27 | **The injectors' instance diversity is measured, and a proposed ADR scopes the fix (issue #86).** Measured over every seed the plan uses (smoke `0,1,2,4`, tune `1000–1015`, eval `2000–2039`; 60 seeds) with real sandboxes: the two measurable faults offer **3 content-distinct environments each — 6 across the family — one per resolution**, and seeds selecting one resolution are byte-identical **including commit SHAs** (`.venv/bin/python -m precondition_library.bench.instance_diversity`). #86's "6 independent observations" is confirmed; its "differing only in commit SHAs" is corrected in the stronger direction — they differ in nothing. The three excluded faults do vary content by seed (`dirty_tree` 2 environments, `branch_renamed` 4, `lockfile_conflict` 56 on the same seeds) but stay outside the comparison because their single fixed sentence is the label (issue #25). **Live corrections:** §10's determinism block said "different seeds -> genuinely different instances", which the measurement refutes for same-resolution seeds and is corrected in place; §7 records the measured instance count and names the proposed direction. ADR-0005 (proposed) decides the approach — parameterise each state shape along declared axes drawn per seed, key occurrence roles on instance identity rather than resolution, restate the seed plan in terms of independent environments, exclude a fault that cannot diversify rather than pad it, and carry the achieved instance count in the report. **This revision scopes the work and changes no injector behaviour**: `tests/test_instance_diversity.py` pins the property as a strict `xfail` naming #86, which fails today and whose marker is deleted by the change that lands it. No benchmark metric, denominator or reported number changes, and no result is claimed. |
| 38 | 2026-09-27 | **ADR-0005 is landed: each measurable fault's state shapes are parameterised along declared Tier 1 axes drawn per seed, independence is keyed on instance identity, and the seed plan's arithmetic is restated (issue #86).** `diverged` and `submodule_moved` now draw, per seed, the file bodies each side writes, the local commit count (1–3) and subjects, and the conflict hunk position; `FaultSpec.instance_for_seed` returns the resolution plus the drawn values, `build_sandbox` records it on the `Sandbox`, and `build_sandbox` re-derives the resolution from the observed `StateFingerprint` and raises if it is not the one `variant_for_seed` declared (the guard is tested firing on a deliberately bad draw). `bench.splits.occurrence_roles` marks an occurrence `variant` the first time an **instance** is seen and `replay` only on a genuine repeat of one; `FaultSpec.variant_for_seed` keeps its meaning. Measured over the 60 plan seeds with real sandboxes (`.venv/bin/python -m precondition_library.bench.instance_diversity`): **61 distinct environments, up from 6** — `diverged` `discard` 23, `merge` 14, `rebase` 8; `submodule_moved` `init` 5, `remove` 6, `repin` 5 — and distinct instance identities build distinct environments; the 40-seed eval set alone builds 47 of them (see §7 for the eval per-resolution table). Every resolution clears the owner's target of 4, and `submodule_moved` reaches it on the nested repo's content alone, so the approved "cannot diversify" exclusion is not applied. The smoke set's four seeds now draw four distinct instances, so it carries **no replay**; the cost curve's replays live in the tune (1 and 9) and eval (8 and 25) sets. `bench.report` carries the achieved instance count per resolution (`achieved_instances`). **Live corrections:** §7's seed-plan table and its "three, not forty" arithmetic are replaced by the measured per-resolution counts and the restated underpower note (a 95% Wilson half-width of about ±18.6pp at the owner's target N=24 and ±14.7pp at the 47 achieved, against the ±10pp equivalence margin's ~94); §7's Design and §12's unit-of-analysis row move from the fault seed/family to the environment; §10's determinism block is corrected in place. The strict `xfail` in `tests/test_instance_diversity.py` is deleted and the test passes for the real reason — two same-resolution seeds now build different file-body signatures — with a sibling test that builds four real environments per resolution. **Logged before any eval pass** — no eval episode has been run, so no analysis was chosen after seeing results; the instance counts are derived from the declared draw and from real sandboxes, and no benchmark metric, denominator or reported result is claimed. |
| 39 | 2026-09-30 | **Revision 38's Wilson half-width at the 47 achieved environments was misstated: it is about ±13.7pp, not ±14.7pp.** `bench.report.wilson_interval` gives a 95% half-width of **0.1374** at 23/47 and **0.1857** at 12/24; ±14.7pp is the half-width at about N=40, the eval set's *seed* count, which is the unit revision 38 replaced with the environment. The error overstated the imprecision. This row corrects only that figure: the ±18.6pp and "~94" beside it are left as written, and whether a single-proportion Wilson half-width is the right yardstick for §7 item 10's TOST on a *difference* of two rates is not settled here. **Live corrections:** §7's underpower note and `bench.splits`'s module docstring now say ±13.7pp; revision 38 is left as written, because a record of a time is not rewritten (rule 5), and this row is the correction. The two quoted half-widths are now pinned by `tests/test_ablation_report.py::test_the_seed_plans_quoted_half_widths`, so a figure that disagrees with the function fails CI rather than a reader. No benchmark metric, denominator or reported number changes, no eval episode has been run, and no result is claimed. |
| 40 | 2026-09-30 | **The two questions #86 left to the owner are settled in ADR-0006 (proposed): `submodule_moved/init` counts as five environments, and the ±10pp equivalence margin stays, with its power restated as the TOST it is.** *`init`:* its five instances are byte-identical before resolution and differ in the nested content their gitlink names, which a correct resolution materialises; the owner reads that content as part of the environment, so the counts stand at 61 plan / 47 eval and `submodule_moved` stays in the comparison. `bench.instance_diversity` now expands each gitlink into the nested commit's blob lines, so no commit id enters `instance_signature` and the five are witnessed by content; the re-measured counts are unchanged (`.venv/bin/python -m precondition_library.bench.instance_diversity`), and `tests/test_instance_diversity.py::test_gitlinks_are_compared_by_the_content_they_name` pins it. *The margin:* revisions 38 and 39 stated the episode loop's power with one rate's 95% Wilson interval, but §7 item 10's margin is applied by a TOST to the **difference** of two arms' rates at a 90% interval. With `bench.report.tost_equivalence` and both arms at a shared 50% rate the difference's half-width is ±22.5pp at 24 per arm and **±16.5pp at 47**, and ±10pp first becomes passable at **133 per arm** (87 at 80%, 53 at 90%), not ~94. The margin is **not** revised: item 10 stands at ±10pp and α = 0.05, and at the achieved N the expected verdict is "comparable", which `success_rate_wording` already produces. **Live corrections:** §7's underpower note and `bench.splits`'s module docstring state the TOST figures beside the single-rate ones, and the ~94 is withdrawn; revisions 38 and 39 are left as written (rule 5). `tests/test_ablation_report.py::test_the_margins_quoted_tost_half_widths` and `::test_the_margin_first_becomes_passable_at_133_per_arm` pin the figures. **Logged before any eval pass**: no eval episode has been run, no metric definition, denominator or margin changes, and no result is claimed. |
| 41 | 2026-09-30 | **The ledger gains `temperature`, and the provider pins it at 0 (issue #6).** #6 asks for the model version and temperature to be pinned and recorded per episode; `model` was recorded, but no temperature was sent or recorded anywhere. `DeepSeekProvider` now sends `temperature` on every call (`DEFAULT_TEMPERATURE = 0.0`, overridable) and reports the value it sent on each `Completion`; `bench/run.py`'s per-episode accounting reads it off the completions, so the row records what the calls used rather than a configuration value, and an episode whose calls disagree raises. A replay row carries `null`, because no model ran. **Not verified:** whether the endpoint honours the field — no live call has been made, a reasoning model may ignore sampling parameters, and temperature 0 is not determinism on a hosted model. The field is additive and defaulted, so no metric definition, denominator or reported number changes; no eval episode has been run, and no result is claimed. |
| 42 | 2026-09-30 | **Arm 2 gets a pre-registered floor: it counts as the baseline only if its informed-regime strict top-1 on the tune set clears chance by its 95% Wilson lower bound (issue #116, ADR-0007).** #116's last open item asked for the top-1 below which arm 2 is not a usable baseline to be decided before the eval, so that "preconditions beat text similarity" cannot be read off a broken one. §7 gains item 11. The measurement is `bench.similarity_probe.tune_baseline` (informed regime, `TUNE_SEEDS`, the eval library's candidate texts), and the verdict is `bench.report.arm2_baseline_floor`: usable when the Wilson lower bound exceeds 1/3. At 24 decidable pairs the floor falls between 12 and 13. Below it the comparison is still reported, with the verdict beside it, and the claim may not be stated. **Context only, not the gate:** hand-written gold on the tune seeds scores 55/96 (57%, AUC 0.6175), lower bound 0.473, and the probe's informed 15/24 has a lower bound of 0.427. Neither is the pre-registered measurement, which needs the compiled library. The probe module stays ungated on `PROBE_SEEDS`; `tune_baseline` is the one gated measurement and refuses an intent with fewer than two candidates, so an empty library cannot read as a failed baseline. **Logged before any eval pass**: no eval episode has been run, no existing metric definition or denominator changes, and no result is claimed. |
| 43 | 2026-09-30 | **The primary metric's arithmetic is defined and implemented: arm 2 is swept, arm 3 is one operating point whose coverage is the pre-registered coverage point, and mismatch is per fire (issue #5, ADR-0008).** §7 said both dispatchers sweep their acceptance thresholds, but arm 3 has no score — it fires the most specific program whose preconditions hold, or nothing — so there was nothing to sweep, and the pre-registered coverage point had no value. The owner chose: arm 3 is a single point and its own coverage is the coverage point; mismatch is wrong fires over fires, with any fire on a negative pair wrong. `bench.coverage` implements it over per-pair dispatch outcomes (`PairOutcome`), so it runs before a compiled library exists: `sweep` (one point per distinct arm-2 score, inclusive floor as in `Library.match_semantic`), `operating_point`, `matched_comparison` (nearest coverage, ties to the lower coverage, compared as exact fractions, gap reported), `vacuous_reason` (arm 3 correct on every pair), and `detectable_difference` for item 3. Every function refuses outcomes from both regimes. **Live corrections:** §7 item 1, item 3, Figure 1 and the method diagram, the README diagram, and the `bench.report` and `bench.pairs` docstrings. **Exercised on gold, not the gate:** over the tune seeds' informed pairs, arm 2 traces 7 operating points from 8/96 to 96/96 coverage (mismatch 41/96 at full coverage, i.e. 55 of 96 correct), and arm 3 played by the labelling rule decides all 96 pairs correctly, so `vacuous_reason` fires, as it does on the 128 uninformed pairs. The first real run needs compiled programs (#4, #7). **Logged before any eval pass**: no eval episode has been run, and no result is claimed. |
| 44 | 2026-09-30 | **The arms can share one frozen library: `bench.build_library` builds it from the admit set, and `run_benchmark(frozen_library=...)` dispatches every arm against it without writing (issue #4, ADR-0009).** Until now each arm grew its own `library-{arm}` online, so the only arm-2-versus-arm-3 number came from different libraries. The build solves each admit-set seed against its own empty scratch library, so the artifact cannot depend on either arm's dispatch, and copies every program -- admitted or rejected -- with its history into one root; its episodes go to their own ledger. The frozen mode suppresses the three online writes (compile on fallback, demotion, quarantine), still records every fire and misfire on the row, and raises if the library's hash changed by the end of the run. The online mode stays the default, for the cost curve, and §8's demotion and quarantine apply there unchanged. **Live correction:** phase 2's exit criterion said both dispatchers sweep a threshold; revision 43 and ADR-0008 made arm 3 one operating point, and the phase table now says so. **Not done:** the library is not built -- that needs a model and a key -- and #4's other items (the 2x2 admission factorial, arm 3-prime, arm 2's top-k rerank, `program_ids` per row) remain open. No eval episode has been run, and no result is claimed. |
| 45 | 2026-09-30 | **The admission factor is runnable: one compile gated twice, with "ungated" meaning positive-only (issue #4, ADR-0010).** `admit(gate=AdmissionGate.POSITIVE_ONLY)` runs the pre-sandbox contract checks and the positive side unchanged and skips only the three negative-sandbox classes, so the factor isolates Claim 3's criterion. `build_library(positive_only_root=...)` stores each compiled program in the two-sided root and gates the same program again into a second root, so the libraries differ only in what each gate admitted; each root's `build.json` records its gate, outside `library_hash`. The ledger gains `admission_gate` (`two_sided`, `positive_only`, or `None` for arm 1 and for a frozen library with no manifest), and `bench.report.admission_factorial` gives one cell per (arm, gate) over graded variant episodes, coverage and per-fire mismatch as in ADR-0008, refusing an unknown gate or a gate spread over two libraries. Arm 3-prime is the {precondition, positive-only} cell. **Live correction:** Figure 3 now says what *ungated* means and where the two libraries come from. **Not done:** the libraries are not built (a model and a key are needed), and no figure is drawn. No eval episode has been run, and no result is claimed. |
| 46 | 2026-09-30 | **Arm 2 can run as retrieve-then-rerank: a second `Similarity` rescores the first stage's top k, and its score is the one thresholded (issue #4, ADR-0011).** #4 asks for arm 2's "top-k rerank before selection"; the representation half (request plus fingerprint text) already existed, the rerank did not. The owner chose a second scorer seam over an LLM reranker, so arm 2's dispatch still makes no LLM call and its replays stay free. `Library(reranker=..., rerank_k=3)` retrieves the top k admitted programs by `similarity`, rescores them with the reranker, and thresholds, orders and records the reranker's score; `None` leaves arm 2 single-stage and unchanged. The rerank lives behind the matcher, not in `dispatch.py`. Its usage is metered into `embedding_tokens`/`embedding_calls`, rows record `rerank_k`, and `bench.coverage.arm2_outcomes` reranks the same way so the pair-level analysis measures the arm the episodes run. **Live correction:** §7's ledger schema lists `admission_gate` (ADR-0010, omitted when that field landed) and `rerank_k`. **Not decided here:** whether the eval runs with a reranker, and which — a tune-pass configuration to be recorded. No eval episode has been run, and no result is claimed. |
| 47 | 2026-09-30 | **Every row names the programs its library held (issue #4).** #4 asks that every ledger row record `library_hash` "and list the program_ids present"; the digest was recorded (#64), the list was not. `EpisodeRecord.library_program_ids` is every program id in the library as the episode found it, sorted -- every status, because the digest covers every status -- and `Library.program_ids()` produces it. It is written on normal and invalid rows alike. The `library_hash` field's docstring still said each arm grows its own library and that removing the confound was tracked separately; since ADR-0009 the frozen mode removes it, and the docstring now says so. The field is additive and defaulted; no metric changes, no eval episode has been run, and no result is claimed. |
| 48 | 2026-09-30 | **The plan runs as whole-run replicates, and seed-level uncertainty comes from an instance-clustered bootstrap instead of a mixed model (issue #6, ADR-0012).** #6 asks for k >= 3 repeats per (fault, seed, arm) and a mixed model with seed as a random effect; neither existed. The owner chose whole-run replicates -- `run_replicates` re-runs the plan from scratch per replicate, with its own ledger and, online, its own libraries, because a back-to-back repeat in one run would replay the first one's program -- and a cluster bootstrap over instances instead of a fitted model, because the repository takes no statistics dependency and a GLMM is unstable at 47 instances. Rows record `replicate`; `cluster_bootstrap` gives arm 2 minus arm 3 per-fire mismatch (ADR-0008) with a seeded 95% percentile interval over 2000 resamples of whole instances, over graded variant rows, skipping and counting resamples in which an arm fired nothing. **Live correction:** §7 item 2 still named the unit of analysis "the fault seed/family"; since ADR-0005 it is the environment, and the item now says so and names the replicates and the bootstrap. Of #6's other items, the seed split, the pairing, the pinned dates, recorded model and temperature (#133) are done, and "occurrences 2-4 as held-out variants, or relabelled replays and excluded" is met by its second branch (`occurrence_roles`); a clean rerun reproducing eval numbers needs an eval run. No eval episode has been run, and no result is claimed. |
| 49 | 2026-09-30 | **Probes must only read: one that changes the sandbox is refused, counted, and fails admission (issue #10).** #10 asks for preconditions that are "pure read-only probes"; since #108 they were screened by the guard, but the guard permits writes inside the sandbox because bodies need them, so a probe such as `git reset --hard` ran and corrupted what every later probe and the episode observed. `evaluate_predicate` now snapshots the sandbox before and after each probe (working-tree files by content, the index as `git ls-files --stage`, `.git/config`, `.git/HEAD`, and the refs of the clone and the upstream; not `.git/index`'s bytes, which `git status` legitimately rewrites) and refuses a probe that changed any part, naming it. `PredicateResult.refused` flags both a guard refusal and a mutation, and `GroundTruthResult.refusals` counts them, so a refusal no longer reads as an ordinary non-match. Admission's negative side treats a refused precondition as a defect, closing the hole where a refused probe rejected every negative sandbox and passed as a program that never fires. §9's threat table gains the row. The snapshot costs about 11 ms (three git calls), twice per probe; the Linux suite's wall clock rose from about 3.6 to 4.8 minutes. **Not done (#10):** the precondition vocabulary and breadth cap, argv templates, process isolation, and the injection suite. No eval episode has been run, and no result is claimed. |
| 50 | 2026-09-30 | **Bodies and probes run in their own process group, killed whole on timeout, under POSIX resource limits (issue #10).** #10 asks that replay run "as an unprivileged user with no network, ulimits, a disk quota, and the process group killed on timeout". `subprocess.run(timeout=...)` killed only the shell, so work it had backgrounded outlived the timeout; measured, a background writer fired after the timeout and wrote its marker. `runtime.confine.run_confined` starts each command in a new session (POSIX) or process group (Windows) and kills the group on expiry (`killpg`; `taskkill /T /F`), and on POSIX sets `RLIMIT_FSIZE` to 256 MiB and `RLIMIT_CORE` to 0. Replay and probes both go through it. §9's structural protections and residual risk are updated. **Not done:** an unprivileged user, network isolation and a disk quota need privilege a process cannot give itself; Windows has no resource limits (job objects not implemented); a body that calls `setsid` escapes the group. No eval episode has been run, and no result is claimed. |
| 51 | 2026-10-01 | **An adversarial injection suite drives hostile programs through the runtime defences and reports containment per tier (issue #10).** #10 asks for a suite that seeds injection "across commit message, filename, branch name, and issue text" and reports "injection-success and containment-breach per defence tier". `bench.injection` plants an attacker instruction as real repository content in each of the four origins and then, because there is no model here to measure steering, assumes the worst case: a `hijacked_compile` stand-in drops that instruction verbatim into a program body or precondition probe, as if the compile step were fully steered. Each measured tier (guard: network, credential/env, outside-write, force-push; hardened-git hooks; the read-only probe check; the process-group kill) plants a canary — a file outside the sandbox, a planted secret, a git hook, a backgrounded writer — and the breach check reads it; the network and force-push tiers, which cannot run without real egress or a real remote, are scored by the guard's refusal. Across the default 2 seeds × 4 vectors × 7 tiers = 56 episodes the suite reports full containment (0 breaches) with injection-success on every episode by construction, and `tests/test_injection_suite.py` pins that its breach detectors fire when a defence is removed, so "contained" is a result and not a detector that can only say yes. §9 gains the suite under structural protections. **Not measured:** the compile-time untrusted-channel tier, which needs a model; it is reported as a row with no episodes rather than silently omitted. **Vector note:** because the stand-in is fully hijacked, containment depends on the tier, not the vector, so the vector axis shows the four origins are exercised, not that origin changes the outcome. No eval episode has been run, and no result is claimed. |
| 52 | 2026-10-01 | **A program may carry argv-template `steps` instead of a shell body, run without a shell (issue #10).** #10 asks the compiler to "select and parameterize only from a fixed allowlist of argv templates ... parameters passed as argv arrays, never shell or f-strings, with per-template type+regex validation and realpath containment". `runtime.templates` adds that path: a `Program` carries either a shell `body` or a list of `steps`, each naming one entry in a closed catalogue (sync: fetch/reset/rebase/merge; submodule: update/checkout/rm/config/add/commit). `runtime.replay` dispatches on which is set; a `steps` program is built into an **argv array** per step from the sandbox's bound values and the step's literals and run with `subprocess` directly, so an attacker-controlled value — a commit message, a `.gitmodules` path — is one element git receives verbatim and cannot become code. Bound values are validated before they reach argv: refs against a charset, paths against a charset and realpath containment under the working tree (so a `.gitmodules` `path = ../../escape` is refused rather than handed to `git -- <path>`). The two hand-written gold resolution sets are migrated to `steps` and run end to end through the executor, so the catalogue is exercised, not a stub; `Program` now requires exactly one of `body`/`steps`, and admission's undeclared-parameter check (#77) reads step parameters for the typed path. §9 gains the argv-template boundary under structural protections. **Not done (#10):** the model-compiled online arms still emit shell bodies behind the guard — migrating the compile step to emit `steps` is issue #4's territory; and the precondition breadth cap and the unprivileged-user/network/disk-quota isolation remain open. No eval episode has been run, and no result is claimed. |
| 53 | 2026-10-01 | **A measured breadth cap: a precondition set that fires on more than half the sampled states is rejected (issue #10).** #10 asks for a "measured breadth cap: predicates matching >X% of sampled states are rejected, and X plus the sampling procedure are recorded." X is `agents.compile.BREADTH_CAP` = 0.50 and the sampling procedure is `sampled_states`: the clean sandbox plus one sandbox per distinct injected state of every fault, at the first seed that selects it (`variant_for_seed`), ten states in all. `precondition_breadth(program)` reports the fraction fired on, and `admit` rejects a program over the cap on the negative side. Measured and grounded, not guessed: every gold program fires on exactly one of the ten (0.10), so the cap sits far above any targeted program and rejects one firing on six or more. The three enumerated negative classes are a sharpening of this holistic cap: after a program is rejected on the clean sandbox, every unrelated fault's states and its declared siblings, only its own fault's remaining states can still fire, and no fault has more than three, so for the current family the cap does not bind — it and the recorded breadth number are the backstop that becomes load-bearing if a fault injects a state no enumerated class names. `admit` computes the cap cheaply from the own-fault remaining states (the enumerated classes already established the rest do not fire) and `precondition_breadth` does the full sweep for any program; `tests/test_breadth_cap.py` pins that the two agree and that a fire-everywhere precondition is measured over the cap and refused. §6 records the criterion. **Not done (#10):** the unprivileged-user, network-isolation and disk-quota items need privilege a process cannot give itself (CI/container work), and migrating the compile step to emit argv-template steps is #4's. No eval episode has been run, and no result is claimed. |
| 54 | 2026-10-01 | **Bodies and probes run with no network where the host allows it (issue #10).** #10 asks that replay run "with no network". `runtime.confine.run_confined` now runs each command inside a fresh, empty network namespace via `unshare --net` when the host can create one, a kernel boundary rather than the guard's textual screen; measured, an egress attempt under it is refused while local-path git (the sandbox's only remote) still works. The privilege to create a namespace may be withheld (a non-root runner, a locked-down container), so `network_isolation_available` probes for it once and, when it is absent, the command runs without the namespace and `Confined.network_isolated` / `ReplayResult.network_isolated` record that it did — the owner's choice: run and record, not refuse (so a host without the privilege, and the Windows path, are no worse off than before). The guard still screens network tools in the body, so the textual screen remains the fallback. §9 gains the mechanism under structural protections and the residual-risk paragraph is corrected: network isolation is kernel-enforced where the namespace is available, and a separate unprivileged user and a real disk quota remain the privilege-gated items #10 leaves open. `tests/test_network_isolation.py` pins egress blocked where available, the recorded fallback everywhere, and that a refused body is never recorded as isolated. No eval episode has been run, and no result is claimed. |
| 55 | 2026-10-01 | **The last two #10 isolation items are recorded as a runner requirement, not library code (ADR-0013).** #10's isolation list ends with a separate **unprivileged user** and a real **disk quota**. Both need a privilege a process cannot give itself — setuid or a user namespace to drop users, a filesystem quota or a size-capped mount for the quota — so implementing them in `runtime` would either assume a privilege the library cannot (as the root-only `unshare --net` already shows) or be a no-op that reads as a guarantee, the "stub that reads like a boundary" this repository rejects. ADR-0013 records the decision: the library enforces only process-giveable isolation, and an unattended eval against untrusted repositories MUST run inside an execution container the operator configures for a non-root user, a size-capped sandbox root, and container-level no-egress (or the namespace capability the library then uses). `docs/running-unattended.md` is the operator's half of the contract — what the library already enforces, a table of what the runner must provide and why the library cannot, and a worked `docker run --user … --network none --read-only --tmpfs` example. The library records `network_isolated` but does **not** verify the user or the quota, and the doc says so plainly so no reader mistakes a green run for a configured runner. §9's residual-risk paragraph now points at ADR-0013 and the new doc. This is documentation of the boundary, not code: no `runtime` behaviour changes, no eval episode has been run, and no result is claimed. |
| 56 | 2026-10-01 | **The first issue #7 baseline lands: a zero-token gold oracle floor (ADR-0014).** #7 asks for four baselines so an arm-3 win is attributable rather than a straw man; the first is "hand-written gold scripts as a zero-token oracle floor". `Arm.GOLD` replays, for each episode, the ground-truth variant's committed resolution (`bench/gold/`, loaded by `bench.gold.load_gold_programs` and injected into the run like the `provider` and the `similarity` seam), making no model call. It is an **oracle** — it reads the state's correct variant to pick which gold program to run — which is exactly why it is a floor: its success bounds what any arm could reach here and its zero cost bounds the cheapest an arm could be. Because it consults ground truth it is excluded from the matched-coverage dispatch comparison (`report.mismatch_comparison` still reads only arm 2 and arm 3); it flows through the arm-generic cost/success tables and the Pareto frontier, where `report` names it the floor, and it cannot misfire (it fires the correct variant by construction). `run_benchmark` refuses the run up front unless the gold set covers every measured fault's every variant (`_require_gold_covers`), so a gap in the oracle is a configuration error rather than a silent dent in the floor. §7 item 4 now lists the oracle floor among the secondary metrics. **Not done (#7):** the soft-classifier (2b), intent-key (2c) and reflexion (1b) baselines remain, each its own change. No eval episode has been run, and no result is claimed. |
| 57 | 2026-10-01 | **The second issue #7 baseline lands: arm 2c, intent-key dispatch (ADR-0015).** #7 asks for "exact/parameterized intent-key dispatch, the cheapest credible competitor": if a lookup on the request string does as well as text similarity or preconditions, the comparison between those two was not measuring what it claimed. `intent_key.intent_key` normalises a request (lower-cased; quoted strings, path-like tokens, commit ids and numbers become placeholders; punctuation dropped), and a program's key is the key of the request it was compiled from (`Provenance.compiled_from_task`). `Library.match_intent_key` returns the admitted programs under the request's key and **abstains** when they implement two or more resolutions -- the owner's choice over "last write wins" (a straw man on phrasings that recur across states) and a majority vote (scored partly on scheduling). `Arm.INTENT_KEY` runs through the same path as arms 2 and 3 -- runtime, guard, checker, fallback, compile, demotion, quarantine, frozen mode -- with no score, so it is one operating point; `bench.coverage.intent_key_outcomes` applies the same rule to labelled pairs. `bench.run`'s arm-to-matcher selection is now an explicit table that raises for an unlisted arm, where a final `else` used to run arm 3's matcher for any arm not named. Measured only as properties, not results: every shipped phrasing of both intents keeps a distinct key, and hand-written gold (whose provenance names no request) fires on none, so 2c is measured on a compiled library. §7 item 4 lists it. **Not done (#7):** the soft-classifier (2b) and reflexion (1b) baselines. No eval episode has been run, and no result is claimed. |
| 58 | 2026-10-02 | **The third issue #7 baseline lands: arm 2b, a soft vote over arm 3's own probe features, and the rule that decides whether Claim 2 is restated (ADR-0016).** #7 asks for "a soft classifier over the SAME probe features with a learned threshold -- if 2b ties arm 3, hard conjunctive executable predicates add nothing over the same information". The owner chose a soft vote: `library.soft_vote_score` is the fraction of a program's preconditions that hold, read from the same `evaluate_preconditions` results arm 3 reads `.ok` from, so arm 3 is the vote at threshold 1.0 and a test pins that the two make identical choices there on real sandboxes. `Library.rank_soft`/`match_soft` and `dispatch_soft_vote` run it behind the library seam; `Arm.SOFT_VOTE` goes through the shared dispatch path, records its score in `dispatch_score` and its threshold in the new `soft_threshold` field, and `run_benchmark` refuses a 2b run without a learned threshold rather than recording a default as if it were tuned. `bench.soft_vote` collects outcomes on real sandboxes (each seed's faulted state as a positive, its clean sandbox as a negative), learns the threshold for most correct decisions with ties to the stricter floor (the owner's choice), and `claim2_verdict` applies the new §7 item 12. **Found while building it:** on hand-written gold at the tune seeds (64 outcomes) the most accurate threshold is **1.0** -- the hard rule -- and lowering it to 0.5 fires on all 64 with 32 wrong, so a naive "equivalent at matched coverage" test would have restated Claim 2 on two arms making identical decisions; item 12 reports that case as **collapsed** and keeps the registered wording. That is an exploratory figure on gold, not the eval. `tost_equivalence` gains a `quantity` label so its reason names mismatch rates when it compares them. **Not done (#7):** the reflexion baseline (1b). No eval episode has been run, and no result is claimed. |
| 59 | 2026-10-02 | **The fourth issue #7 baseline lands: arm 1b, ReAct with a memory of its own past successes (ADR-0017).** #7 asks for "react plus prior-success memory/reflexion, the standard experience-reuse baseline whose absence makes the react curve artificially flat". The owner chose trajectories over written reflections and the model's own verdict over the checker's: `agents.memory.SuccessMemory` stores the request and the git commands that ran without error, written when `solve` returns `SUCCESS` (the model declared the task done), inside the arm and before `run_episode` runs the fault's checker, so no ground truth reaches it and a declared-but-wrong episode is remembered as a deployment's would be (a test pins exactly that). Each episode recalls the `RECALL_K = 3` most lexically similar entries (ties to the more recent; zero-similarity entries never recalled), rendered as a prelude to the **user** message with a warning that they may not apply, so the system prompt every arm sends is byte-identical and the prompt-prefix report stays true. No extra model call is made; 1b's extra cost is the longer prompt, charged through the same accounting. `Arm.REACT_MEMORY` keeps one memory per arm per run, records `memory_recalled` on its rows, has no library and no admission gate, and is refused in a frozen run, which exists so that nothing learns mid-run -- 1b is a baseline for the online cost curve. An end-to-end test runs arm 1 and 1b on the same plan and pins that 1b's second episode carries its first episode's commands while every earlier prompt carries none. With this row all four #7 baselines exist. No eval episode has been run, and no result is claimed. |
| 60 | 2026-10-02 | **Issue #4's last test item: only the dispatch function varies between arms, and admission is the only free variable (ADR-0018).** #4 asks for "tests [that] assert all arms load the same library_hash and that admission settings are the only free variable"; one test covered arms 2 and 3, and arms 2b and 2c (#153, #152) dispatch from the library too. `bench.run.DISPATCHING_ARMS` now names the arms that choose from a library (2, 3, 2b, 2c), `_dispatch` serves exactly that set, and the `admission_gate` field is derived from it, so arm 1, 1b and the gold oracle record no gate by construction rather than by a hand-kept list. `tests/test_one_free_variable.py` pins three things: the dispatch table serves exactly `DISPATCHING_ARMS` and refuses every other arm; in one frozen run every dispatching arm reads the same unchanged artifact (one hash, one program-id list, one gate, all replaying for zero tokens); and two frozen runs whose libraries differ only in the gate their build recorded produce rows that differ in `admission_gate` and in no other field the runner writes (wall-clock aside), so the 2x2's admission factor carries no confound from the harness. It also settles the design choice #4's first comment left open: `StateFingerprint.observe` reads `upstream/main`, so every fault must leave it resolvable (ADR-0018, written on `FaultSpec.inject`), and the test builds every state in `sampled_states()` -- admission's own negative universe -- and asserts the ref resolves and `observe` returns. **Not done (#4):** the frozen library is not built or committed and the factorial has not been run; both need a model and a key. No eval episode has been run, and no result is claimed. |
| 61 | 2026-10-02 | **The primary metric is reported, not only computed: Figure 1, the matched comparison and the power statement (issue #5).** #5's open items were the swept mismatch-vs-coverage curve "with Wilson confidence intervals at each operating point" and "an explicit power statement". `bench.coverage` already held the arithmetic (revision 43, ADR-0008), but nothing assembled it: no call site produced the curve, the intervals or item 3's statement. `bench.primary.primary_report` now does, for one regime at a time (refusing pooled outcomes, ADR-0004, and outcome lists from different pair sets): arm 2's `sweep` with a 95% Wilson interval on every point, arm 3's single point as the pre-registered coverage point, `matched_comparison` against it, and a power statement from `detectable_difference` at the matched fire counts -- always at a conservative 50% base rate (the least-powered case, as ADR-0006 stated its margin's power) and also at the observed pooled rate when that rate is informative. The vacuity guard from #5's design comment comes first: while `vacuous_reason` fires, `summary` prints it **instead of** the comparison and the figure's title says so. #7's dispatching baselines may be added to the plane as context, a scored one as a curve and an unscored one as a point. `write_primary_report` writes `figure1.csv` (the deliverable), `primary.txt` and `figure1.png`; Figure 1's paragraph now names it. Both arms of #5's "threshold swept for both dispatchers" are met as ADR-0008 re-defined them: arm 2 is swept, arm 3 has no score and is one point. **Not done:** the first real Figure 1 needs a compiled library's pair outcomes (#4). No eval episode has been run, and no result is claimed. |
| 62 | 2026-10-02 | **The provider sets its own timeout, and a transport failure is a retried, counted provider error (issue #157).** The first live run (#163) lost every compile call to httpx's **5-second default timeout**: `DeepSeekProvider` built its client with none, a compile reply from a reasoning model takes longer, and the build admitted nothing until the run's own driver raised the limit. `DeepSeekProvider` now takes `timeout` and `connect_timeout`, defaulting to `DEFAULT_TIMEOUT_S = 300` and `DEFAULT_CONNECT_TIMEOUT_S = 30` -- the values that smoke build completed every call under -- and wraps any `httpx.TransportError` (a timeout, a refused or reset connection) as `ProviderTransportError`, a `ProviderError` with no status, as `ProviderError`'s docstring had always described transport failures. `bench.run._is_retryable` treats it as transient and retries it under the same three-attempt cap as a 429 or 5xx; a malformed body, the other status-less failure, stays unretried. Because it is now a `ProviderError`, `_AccountingProvider` counts every timed-out attempt in `llm_calls`, which it previously could not see. §8's degradation table now says so. No eval episode has been run, and no result is claimed. |
| 63 | 2026-10-02 | **A per-invocation git config override can no longer undo the hardening (issue #159).** The first live run (#163) found, and a reproduction then verified, that `git -c core.hooksPath=<dir>` runs hooks from `<dir>` under the hardened environment, because command-line config wins over the `GIT_CONFIG_COUNT` pins; the replay guard allowed such a body, and the ReAct tool's `run_git` let the live agent use `-c` freely. `sandbox.git_config_overrides` now parses git's **global** options only -- so `git log -c` (a combined diff) is not mistaken for config -- across every nested `git` in a command, including a `submodule foreach` command given as one quoted string; `disallowed_git_config_overrides` compares them case-insensitively against `ALLOWED_GIT_CONFIG_OVERRIDES`, an allowlist of exactly `protocol.file.allow=always` (the local-file transport the submodule family needs), with `--config-env` never allowed because its value cannot be vetted. The guard refuses any disallowed override and any body that sets a `GIT_CONFIG*` variable in its own environment (which would replace the pins outright); `run_git` refuses the same overrides. A persistent repository `git config core.hooksPath` write was verified **not** to beat the hardening and is left alone. The injection suite gains a `git_config_override` tier -- a hook committed as ordinary work-tree content and a body pointing git at it -- contained by the guard, with a control test showing the same command fires the hook under the hardening alone. §9's threat table gains the row. No eval episode has been run, and no result is claimed. |
| 64 | 2026-10-02 | **Admission judges an overlap state by the program's own intent, not as unrelated (issue #158, ADR-0019).** The first live build (#163) rejected every compiled diverged program on `lockfile_conflict` seed 0, counted as an *unrelated* state where any fire is a defect -- but that state is a diverged branch whose two sides touched one file, and the sync intent's own `decided_by` labels it **merge**. The negative set held a state the task family's own labels call correct for a merge program, so the gate rejected correct programs; the hand-written gold merge program passed only through the `local_work_beyond_the_overlap` precondition, added earlier to dodge this exact state. Owner's choice: `admit` now labels each unrelated-fault state with this program's intent (`_preconditions_accept_labelled`, one sandbox build for both the label and the probes). A state the intent labels is an **overlap state**, judged like a sibling: a fire is correct when the program implements the label, a mismatch otherwise; only an unlabelled state stays unrelated, where any fire is a defect. Correct overlap fires count toward the breadth cap. `tests/test_overlap_states.py` pins the overlap itself, that gold merge minus the workaround precondition is now **admitted**, that the same program labelled `rebase` is **rejected** with the overlap reason, and that gold merge still passes. The negative-class diagram in §5 says so. The submodule intent's rules are also fixed here, because this decision depends on them: they labelled a repository with no submodule `init` -- `observe` reports a missing submodule as not initialised and still referenced upstream -- which made every diverged state and the clean sandbox an `init` overlap state for a submodule program. Each rule now first requires `has_submodule_reference`; every declared grid state already has one, so no grid label moves. No eval episode has been run, and no result is claimed. |
| 65 | 2026-10-02 | **The agent's budget counts commands as well as turns, rows record tool calls, and only a declared success is compiled (issue #160, ADR-0020).** In the first live run (#163) the model issued 4-5 tool calls per turn, so the 12-turn `max_steps` let one episode run about 45 commands and another 12, depending only on batching; all four submodule solves used up the budget, and those FAIL transcripts were still compiled -- two admitted. Owner's choices: `agents.react.solve` keeps the 12-turn budget and adds `DEFAULT_MAX_TOOL_CALLS = 24` (twice the turn budget, so a one-call-per-turn agent never meets it first), stopping with `FAIL` once that many calls have run however they were batched; every row records `tool_calls`, refused calls included, beside `llm_calls`; and `_run_arm` marks a fallback `compilable` only when the agent itself returned `SUCCESS` -- the agent's verdict, which the no-oracle rule allows, not the checker's. `tests/test_tool_call_budget.py` pins the cap against one 30-call turn, the recorded count on a batched solve, and that a failed solve triggers no compile call. §7's ledger schema lists `tool_calls`. No eval episode has been run, and no result is claimed. |
| 66 | 2026-10-02 | **The harness keeps no bookkeeping in the clone (issue #161, ADR-0021).** The first live run (#163) found the ReAct agent reading `refs/sandbox/submodule-path`, `refs/sandbox/refs-at-start` and `refs/sandbox/injected` -- the three refs #103 had left in the clone. A precondition anchored on one holds in every sandbox and in no real repository, and `refs-at-start` is a ready-made diff of the injection. All three move into `Sandbox.recorded`: `require_uninjected` reads the recorded `base`, reversing revision 24's on-disk marker (a `Sandbox` is only built by `create`, so the in-memory record lives as long as the sandbox); `submodule_moved` records `RECORDED_SUBMODULE_PATH`, and `submodule_path` now takes the `Sandbox`; `tasks.invariants` holds `RECORDED_REFS_AT_START`. `store_blob` has no caller left and is removed. `recorded_state_intact`'s "every `refs/sandbox/*` ref still resolves" clause, which doubled as the re-clone detector, becomes "every object a work ref named at the start is still in the object store"; that is weaker for a fault whose starting commits are all on upstream (`dirty_tree`), where the fault's own checker still catches the loss. `tests/test_harness_record_outside_clone.py` pins that no fault leaves a `refs/sandbox/` ref in either repository at seeds 0 and 1, that a second injection is still refused, and that `{submodule_path}` still binds after a correct removal; `tests/test_non_destructive_invariants.py` pins the re-clone violation and a `reset --hard` control. No committed content changes, so no commit SHA moves. No metric definition, denominator or reported number changes and no result is claimed. |
| 67 | 2026-10-02 | **Figure 1's pair outcomes come from a compiled library's own matchers on real sandboxes (issue #162, ADR-0022).** The first real Figure 1 (#163) needed a driver outside the repository, because `coverage.arm2_outcomes` takes one text per variant and `arm3_outcomes` a caller-supplied decision. `bench.library_pairs.library_pair_outcomes(library, faults, seeds)` builds each pair's sandbox and returns one `PairOutcome` list per dispatcher, index-aligned with the pairs: arm 2 through `match_semantic` with no floor (the library's configured threshold is not applied, so `sweep` sees every score), arm 3 through `match_preconditions`, 2b through `rank_soft` with its score, and 2c through `match_intent_key`; the signature is built as `bench.run` builds it, the request is the uninformed one the episodes send, and the library is never written. Each measured intent's request is paired with its own fault's state, the fault-free sandbox, and another fault's state only when the intent labels it. An unlabelled foreign state is not a pair (`is_pair`): the first version included those as negatives, and arm 3, which reads only the environment by design, fired the sync program on diverged states under the submodule request, which no episode poses. Building the pairs also found that the submodule intent labelled a repository with no submodule `init`; that is fixed in #158's change, revision 64. `arm2_outcomes` and `arm3_outcomes` stay for the declared grid and the gold probe. `tests/test_library_pairs.py` pins the pair set, that gold arm 3 is the labelling rule (so `vacuous_reason` fires), that arm 2 is unthresholded under a 0.9 library floor, that a candidate program never fires, that 2c fires on a program compiled from the pair's request, the inclusion rule on its own, and that the result feeds `primary_report`. No eval episode has been run, and no result is claimed. |
| 68 | 2026-10-02 | **A live run is one committed command (#163).** The first live run went through a driver outside the repository, so the run that found #157-#162 could not be repeated from what is committed. `bench.live` is that driver, committed. `python -m precondition_library.bench.live --out DIR` runs, in order: `build_library` on the smoke seeds into a two-sided and a positive-only root (ADR-0009/0010); `learn_soft_threshold` on the tune seeds; `run_benchmark` with every frozen arm against the two-sided root, and arms 2 and 3 against the positive-only root; `write_report` on the two-sided ledger only, so the main tables never mix two libraries' rows for one arm; `admission_factorial` on the two ledgers concatenated; `library_pair_outcomes` and `primary_report` for Figure 1 on the eval seeds, with 2b's `claim2_verdict`; and, unless `--skip-online`, arms 1 and 1b online. The output directory must not exist. `summary.json` and `summary.txt` record the plan, the admitted counts, the library hash, the learned 2b threshold, the Claim-2 wording, Figure 1's summary, the factorial cells and every artifact's path. The key's source is named by the caller, `--api-key-env NAME` or `--api-key-file PATH`, exactly one, never the key itself, and nothing prints or writes it; the package still finds no key on its own. `--episode-seeds` defaults to the first live run's 4 eval seeds. `tests/test_live_run.py` replaces the two model-calling stages with recording stand-ins and runs the rest for real on admitted gold: the stage order, the library each benchmark ran against, that 2b runs at the learned floor, that 1b is online only, that the report reads one ledger, Figure 1's vacuity on gold, the summary files, the fresh-directory refusal, and the key handling. No live run has been made with it, and no result is claimed. |
| 69 | 2026-10-02 | **A live run whose build admits nothing stops cleanly and says why (#173).** The second live run's build admitted none of its three compiled programs (#171, #172), so `bench.live` reached the frozen benchmark, `run_benchmark` refused the empty library, and the command exited with a traceback and no summary. Now the summary is always written, and every summary carries `build`: one entry per build episode with its outcome, tool calls, the checker's verdict, whether it was admitted, and the gate's `compile_failure_reason`, or `not compiled: the agent did not declare the task done` for a FAIL that #160 kept out of the library. When the two-sided library admitted nothing, the 2b tuning, the frozen benchmark, the factorial and Figure 1 are skipped, because they have nothing to dispatch. `stopped_after` is `build` with a stated `stop_reason`. The online arms 1 and 1b still run, since they need no frozen library. The command then prints the summary and exits 1 with a one-line message. `tests/test_live_run.py` pins the per-episode build lines, the skipped stages, that the online arms still ran, the summary files, and the exit code. No live run has been made with it, and no result is claimed. |
| 70 | 2026-10-02 | **A mismatch is a fire outside the resolutions a state accepts, not a fire other than its label (issue #172, ADR-0023).** The second live run (#163) found the model rebasing successfully on `merge`-labelled diverged states, and every resulting `rebase` program rejected as a mismatch. The labels named one resolution per state; the checker grades the outcome, not the method. Owner's choice: **multiple** acceptable resolutions. Measured before the change, by replaying every gold body on every state over 60 seeds: on `diverged`, merge and rebase pass the checker in every state, and discard passes only where it is the label; on `submodule_moved`, exactly one passes per state, and it is the label. `ResolutionVariant.accepted_by` declares where else a resolution is acceptable, and `IntentSpec.acceptable_variants(state)` returns the label plus those, or `()` for an unlabelled state. `program.accepts` is the one rule every scorer applies: the pair metric (`LabelledPair` and `PairOutcome` carry the set, which drives `operating_point`'s mismatch and `vacuous_reason`), the 2b tuning, the ledger's `misfired` (rows record `acceptable_variants`; an old row without it reads as the label alone), online demotion's mismatch count, and admission's sibling and overlap classes, where a fire on a state that accepts the program's resolution is correct and counts toward the breadth cap. **Consequence:** on `diverged` the only wrong fire left is `discard` on a state with real local work, so the intent tests whether a dispatcher avoids the destructive resolution, not which of three it picks; `submodule_moved` is unchanged and remains a three-way choice. The text-similarity probes (`bench.similarity_probe`, `bench.textcontrol`) and their recorded baselines (ADR-0003, ADR-0004, ADR-0007) keep scoring against the single label: they measure whether text predicts the label, and nothing they report is recomputed. `tests/test_acceptable_variants.py` replays every gold body on every injected state and requires the checker's verdict to equal the declared set; the tests that pinned `rebase`-on-`merge` as wrong now pin `discard` on a `merge` state instead. No eval episode has been run, and no result is claimed. |
| 71 | 2026-10-02 | **The agent declares done with a `finish` tool, and is reminded once of its budget (issue #171, ADR-0024).** In the second live run (#163) the model never ended a turn without a tool call -- the only declaration the loop recognised -- so 5 of 8 build episodes failed, 3 of them with the checker already passing, and since #160 compiles only declared successes, no `submodule_moved` program was compiled. `agents.react.solve` now accepts a call to `finish` (`FINISH_TOOL`, with a one-line `summary`) as the declaration: the solve returns `SUCCESS`, calls batched before it in the turn run and calls after it do not, and it is not a command -- it does not run, is not a tool result, does not count toward `tool_calls`, and is accepted with the tool-call budget spent. A turn without a tool call still ends the solve. `BUDGET_NUDGE`, one fixed user message about budget and nothing about correctness, is sent once before the turn that may be the last: the final turn, or `NUDGE_TOOL_CALL_MARGIN` (4) or fewer tool calls left. The system prompt names both tools and says to call `finish` as soon as the request is satisfied. Every arm's fallback is this agent, so the arms change together; the longer system prefix is what `prompt_prefix_lengths` reports. `tests/test_finish_tool.py` pins the declaration, the batching rule, `finish` past a spent budget, the reminder's single placement by turns and by tool calls, its absence on a short solve, and that its text names no verdict; `test_react_baseline`'s step-budget transcript gains the reminder. No live run has been made with it, and no result is claimed. |
| 72 | 2026-10-02 | **Every agent solve's transcript is kept beside its ledger row (issue #171's last item).** The ledger stores numbers, not what the agent did, so diagnosing #171 -- an agent that fixed the repository and kept verifying until its budget ran out -- needed a separate traced re-run. `run_episode(transcript_log=...)` now appends one JSON line per solve, carrying the row's identifying fields (arm, task, fault, seed, occurrence, replicate, outcome) and the transcript exactly as `solve` returned it; an episode in which no agent ran (a replay, the gold oracle) writes nothing. `run_benchmark` and `build_library` write it by default to `<ledger stem>.transcripts.jsonl` beside their ledgers (`ledger.transcripts_path`), flushed per line like the ledger, so a killed run keeps what it finished, and `bench.live` lists the build's. A transcript holds the system prompt, the request, the commands and their output: sandbox paths, never the provider's key. `tests/test_transcript_log.py` pins the path, the keyed line of a solve that finishes with `finish`, that the oracle writes nothing, and the default sidecar of `run_benchmark`. No metric changes, and no result is claimed. |
| 73 | 2026-10-03 | **`bench.live` runs end to end in the suite, with only the model scripted (#170's weakest point).** `tests/test_live_run.py` stubs the build and the episode runs, so the first live run was also the first proof that the real stages compose. `tests/test_live_end_to_end.py` runs `run_live` with every stage real -- `build_library`, admission, the 2b tuning, both frozen runs, the factorial, the report, Figure 1, the online arms and the transcript sidecars -- on real sandboxes, against a scripted model that, as the agent, calls `finish` at once and, as the compiler, reads `state_at_arrival` from the compile payload, asks the intents' own rules which resolution it needs, and replies with that resolution's committed gold program. It found a defect the stand-ins could not: with the online arms on, the summary never listed `online_ledger`, because `run_live` wrote it into a local dict that pydantic had copied when building `LiveSummary`; it is now written into `result.artifacts`. The test pins that the build compiles and admits real programs, every artifact is written, every frozen arm ran against one library hash with `acceptable_variants` recorded, transcripts are kept for every solve, and gold arm 3 never mis-fires in Figure 1. No metric changes, and no result is claimed. |
| 74 | 2026-10-04 | **A shell variable a body assigns itself is not an environment read (issue #180).** In the third live run the compiled `submodule_moved` program from build seed 1 was refused at admission for `$a`, in a probe that assigned `a=$(git rev-parse ...)` one statement earlier; the guard's environment-read rule matched any `$NAME`. `runtime.guard._env_read_effect` now skips a variable the same body assigned in an earlier statement -- `NAME=...` (with `local`, `export`, `readonly`, `declare` or `typeset`), `read NAME...`, `for NAME in` -- counting the assignment from the end of its statement, so `FOO=$FOO:x` and `a=1 cmd "$a"` still read the environment and are still refused, as are unassigned variables, a read before the assignment, `env`/`printenv`, the `os.environ`-style expressions, and any credential-looking name (the credential rule is separate and unaffected). The scan stays lexical: an assignment inside a `$( )` subshell is counted although it does not reach the parent shell, which can let through only a read of a name the body itself chose. The refusal a probe records no longer doubles its verb (`refused refused ...`). `tests/test_guard_shell_variables.py` pins the allowed forms, the still-refused ones by the variable named, and the live probe evaluating -- not refused -- on a real `repin` sandbox. No metric definition changes, and no result is claimed. |
| 75 | 2026-10-04 | **When a budget runs out, the agent gets one last word with only `finish` on offer (issue #179, ADR-0025).** In the third live run two build episodes fixed the repository with their last batch after the reminder, then overran the 24-call cap with a verification batch before calling `finish`, so neither was compiled. `agents.react.solve` now answers every call batched past the cap with a refusal marked `not_run` (not executed, and not counted by `bench.run._tool_calls`), and when either budget is spent -- the tool-call cap or `max_steps` -- sends `LAST_WORD` and makes one more model call offering `finish` alone. Only a `finish` call there is `SUCCESS`; a plain reply, any other call or a provider error is `FAIL`, recorded with the budget that ran out. A turn without a tool call still ends the solve on every other turn. `tests/test_finish_tool.py` pins the rescue, that the last word offers only `finish`, that a command there is not a declaration, and that the turn budget ends in a last word too; the budget tests now script the last word and count only commands that ran. No live run has been made with it, and no result is claimed. |
| 76 | 2026-10-04 | **The live run measures item 11's arm-2 floor and reports it beside Figure 1 (issue #184).** The floor's pieces existed (`tune_baseline`, `arm2_baseline_floor`), but no run called them, so the third live run reported its matched difference with no verdict on its baseline. `bench.live.baseline_floor` now measures it before any eval episode, exactly as registered: strict top-1 by label, informed regime, `TUNE_SEEDS`, against one text per resolution from the two-sided library's admitted programs. An intent the library leaves with fewer than two candidate resolutions -- the third run's `diverged`, compiled to `rebase` only -- is **not measurable, so not usable**, the case ADR-0007 already rules on (no decidable pair is not usable). Below the floor the summary says Claim 2 may not be worded as a win over text similarity; the comparison is still reported. The registered definition is unchanged: whether it should follow ADR-0023's acceptable sets is an open question for the owner, not part of this row. |
| 77 | 2026-10-04 | **The live run runs item 2's replicates and reports the clustered interval (issue #185).** Item 2 registers the episodes as k = 3 whole-run replicates, and `run_replicates` existed, but `bench.live` called `run_benchmark` once per stage, so the one command could not run the registered plan. `--replicates K` (`LivePlan.replicates`, default 1 for a smoke run) now sends the frozen, positive-only and online stages through `run_replicates` when K > 1: each replicate writes `<stage>/replicate-r/ledger.jsonl`, the frozen libraries are shared and each online replicate grows its own, and the stage's ledger is every replicate's rows concatenated, each keeping its `replicate`, so the report and the factorial read them all. K = 1 keeps the earlier layout. Whatever K, the summary carries `cluster_bootstrap` over the frozen rows (ADR-0012), which `write_report` does not compute, so the episode comparison is reported with instance-level uncertainty rather than as pooled rows. Figure 1 is pair-level and unchanged. The registered plan is `--episode-seeds 40 --replicates 3`. |
| 78 | 2026-10-04 | **Parameter bindings can be derived from an existing checkout (issue #181, first item).** `runtime.probes.bindings` supplies the harness's fixed vocabulary (`upstream`, `main`, the recorded submodule path), so a compiled program could bind only inside a sandbox `sandbox.create` built. `repository_bindings(work)` reads the same names from a repository: `upstream_remote` is `upstream` if present, else the remote the current branch tracks, else the only remote; `upstream_branch` is that remote's recorded `HEAD`, else `main` or `master` if it has one; `submodule_path` is the first path `.gitmodules` declares. A value the repository does not settle is absent, never guessed, so `substitute` reports it unbound and a program needing it is inapplicable. The harness keeps `bindings`, and on every harness sandbox (fault-free and each measured fault's states) the two agree. Nothing calls it yet: the checkout environment, the dry-run and the CLI are #181's later items. No measurement changes. |
| 79 | 2026-10-04 | **An existing checkout is an environment the runtime accepts (issue #181, item 2).** `checkout.open_checkout(work)` returns a `Sandbox` whose `work` is the user's repository and whose `root` is a scratch directory, with a `CheckoutContext` carrying what differs from a harness sandbox. The owner chose a **trusted pre-fetch**: the tool runs one fixed `git fetch <remote>` with the user's own git, mirrors the fetched remote-tracking refs into a local bare repository sharing the checkout's objects, and every model-authored probe, body and step then runs under `sandbox.run_env` -- the harness's hardened environment plus `url.<mirror>.insteadOf <remote URL>` -- so a program's `git fetch` reads the mirror with no network and its `git push` reaches only the mirror. Commits carry the user's `user.name`/`user.email`, dated now, never the sandbox's pinned identity and date. Parameters are `repository_bindings`' (revision 78), returned by `bindings` for a checkout so probes and bodies still share one mapping. `destroy` removes the scratch root, never the checkout. **Not isolated, and stated:** a replay changes the user's repository, so a caller must confirm first; and a probe that writes is refused but not undone, so a dispatcher must probe a copy of the checkout. Harness sandboxes are unchanged: `run_env` is `git_env(home=root)` when there is no checkout. No measurement changes. |
| 80 | 2026-10-04 | **The library dispatches on an existing checkout and reports why (issue #181, item 4, report half).** `dispatch.dispatch_report(library, work)` opens the checkout (revision 79), probes a **copy** of it (`checkout.snapshot`: the whole tree with its `.git`, so uncommitted state is still read, under the scratch root), because a probe that writes is refused but not undone, and reports every admitted program with each precondition's result and the one arm 3 would fire: the first whose every precondition holds, most specific first, `Library.match_preconditions`' rule. The request is recorded and not read, since arm 3 reads only the environment. `python -m precondition_library dispatch --repo PATH --library DIR [--request TEXT] [--no-fetch] [--json]` prints it. **Nothing is replayed**; replaying on confirmation is the next item. A linked worktree is refused, because its copy would share the original git directory. On every harness state treated as a checkout the report fires exactly what arm 3 fires on the sandbox (`tests/test_dispatch_checkout.py`). No measurement changes. |
| 81 | 2026-10-04 | **Dispatch on a checkout dry-runs on a copy and replays only on confirmation (issue #181, last item).** `dispatch.run_dispatch(..., confirm=...)` takes the report of revision 80, replays the program that would fire on a **second, fresh copy** of the checkout, and reports what it did there -- the commits it added, the working tree's status, which `sandbox_state` parts moved, whether its postconditions held. The caller's `confirm` sees the report and the dry run; only on `True`, only if the dry run succeeded, and only if the checkout's `sandbox_state` is unchanged since the copies were taken, is the program replayed on the checkout itself. `python -m precondition_library dispatch --replay` asks `Replay <program> on <repo>? [y/N]` after printing both; `--yes` confirms without asking. This is the dry run `runtime.guard.prepare_dry_run` describes (run the body against a copy and diff), placed in `dispatch` because only it can copy a checkout; the stub stays for a harness-side consumer. Tested on harness states treated as checkouts: declining leaves the checkout's state identical; confirming replays and the fault's own checker passes; a checkout edited after it was copied is not replayed on. No measurement changes. With this, #181's four items exist. |
| 82 | 2026-10-04 | **The fingerprint reads the environment's own upstream ref (prerequisite for #188).** `StateFingerprint.observe` compared every count, touched-file list and submodule probe against the literal `upstream/main`, the harness's fixed spelling of upstream. On an existing checkout upstream is its derived `<remote>/<branch>` (revision 78), a fork's `origin/trunk`, so `observe` now asks `sandbox.upstream_ref(env)`: `upstream/main` for every sandbox `create` built, the checkout's derived ref otherwise, and `NoUpstreamError` for a checkout that settles no upstream rather than comparing against a ref that is not its upstream. **Nothing measured changes**: on the fault-free sandbox and one seed per state of each measured fault, the sandbox observed directly and observed as a checkout give identical fingerprints (`tests/test_fingerprint_upstream_ref.py`). |
| 83 | 2026-10-04 | **On a checkout the agent's git tool runs confined and through the mirror (prerequisite for #188).** The ReAct agent's `run_git` tool called git directly with the harness environment, which is right for a sandbox -- its remote is a local path, and every measurement ran that way -- and wrong for an existing checkout, whose remote is real: a model-authored `git fetch` would reach the network, which the owner ruled out for anything model-written (revision 79). On a checkout (`env.checkout` set) a tool call first passes `runtime.guard.screen`, the screen every replayed body passes -- explicit URLs and `git@host:` targets, credential paths, writes outside the checkout and force-pushes are refused before anything runs, which on a host with no network namespace is what stops `git push https://...` from carrying the user's code away -- then runs under `sandbox.run_env`, so the upstream remote's fetch and push URLs read the trusted pre-fetch's mirror, **every other remote's URLs lead to a path that does not exist** -- in a fork `origin` is the user's own repository, and a solve on a disposable copy must not push to it -- and a commit carries the user's identity, and through `runtime.confine.run_confined`, so there is no network where the host allows that. The tool's own refusals still come first. **The harness path is unchanged**: a sandbox `create` built still calls `run_git` directly, unscreened (`tests/test_agent_tool_on_checkout.py`). |
| 84 | 2026-10-04 | **Learning on a checkout is decided, and learned programs are marked (issue #188, ADR-0026).** ADR-0026 records how a program is learned on an existing repository: solve a disposable copy with the agent confined, the caller names the family, the user confirms what the solve did, and the program must pass the harness's two-sided gate unchanged and also fire and satisfy its postconditions on a fresh copy of the checkout. This revision lands the marker: `Provenance.learned_on` is `"checkout"` for a learned program and `None` for a harness one, **omitted from the serialized form when `None`**, so every harness program's canonical content and every library hash a ledger recorded is unchanged; `bench.run` refuses a frozen library holding a checkout-learned program, because its admission was partly a person's judgement. The learn command itself is the next change. No measurement changes. |
| 85 | 2026-10-04 | **A program can be learned on an existing checkout (issue #188, ADR-0026).** `learn.learn(library, work, request, fault, provider, confirm)` and `python -m precondition_library learn --repo PATH --library DIR --fault FAMILY --request TEXT (--api-key-env NAME | --api-key-file PATH) [--yes] [--no-fetch] [--json]` implement ADR-0026 in order: open the checkout and solve a **copy** (`snapshot(env, "solve")`) with the agent screened, confined and mirrored (revision 83), fingerprinted against the checkout's own upstream (revision 82); stop unless the agent declared done; show the user what the solve did to the copy (commits added, working tree left, parts moved) and stop unless they confirm; compile with the family's declared resolutions; require every precondition to hold, and the replay to satisfy its postconditions, on a **fresh copy of the checkout as it was**; run `admit` **unchanged** on harness sandboxes of the family, at the first seed whose injected state is the program's own resolution (the positive side the harness gives a program compiled there); and only then store it admitted, marked `learned_on: checkout` (revision 84), renamed on an id collision. Each stop records why. Tested end to end with a scripted model on a merge-labelled `diverged` state copied out into a standalone repository: a confirmed solve is learned, admitted, marked and then fires on dispatch, and the repository is untouched; a declined one compiles nothing; a wrong resolution stops before the gate; a program that holds everywhere passes the checkout checks and is still refused by the harness gate (clean sandbox). No measurement changes. |
| 86 | 2026-10-04 | **The fingerprint observes uncommitted files and untracked collisions, unrendered (issue #189, step 1).** The dirty-tree intent #189 proposes decides between resolutions by two facts `StateFingerprint` did not have: `dirty_files`, the tracked files with uncommitted changes (`git diff --name-only HEAD`, staged or not), and `untracked_upstream_collisions`, the untracked files at a path upstream's tree holds -- the state where neither stash-sync-pop nor commit-then-merge keeps the work. Both are observed by `observe` against the environment's own upstream ref (revision 82). **Neither is rendered by `as_text`** (`_NOT_YET_RENDERED`): `as_text` is arm 2's whole view of state, so a rendered field changes every arm-2 score in every measured family, and rendering these is part of the change that registers the dirty-tree intent, with the owner's sign-off. Until then every measured arm-2 text is byte-identical. No measurement changes. |
| 87 | 2026-10-04 | **The dirty-tree fault injects three states that separate its resolutions (issue #189, step 2).** Each `dirty_tree` seed now selects `disjoint` (an uncommitted edit in a file upstream left alone, plus an untracked file), `same_file` (an edit in the file upstream changed, a separate hunk) or `collision` (an untracked file at a path upstream's new commit adds). Measured with plain git before building: stash-sync-pop and commit-then-merge keep the work on the first two and both fail on a collision -- the pop cannot restore an untracked file over a tracked one, the merge meets an add/add conflict -- while moving the colliding file aside first keeps both. The checker is unchanged except that, on a collision, the untracked work may be at `<path>.local`, the one place it can survive beside upstream's file. `variant_for_seed` stays `None`, so admission still builds one dirty_tree state as an unrelated negative -- seed 0, now `same_file` -- and no measured program accepts it (local-only commits or a submodule are what they require). The fault stays excluded. No measurement changes. |
| 88 | 2026-10-04 | **The dirty-tree intent is defined, and its acceptable sets are pinned to the checker -- not yet registered (issue #189, step 3).** `tasks.faults.dirty_tree.INTENT` (`keep_uncommitted_work_and_sync`) labels `disjoint` `stash`, `same_file` `commit`, `collision` `aside` and a clean tree nothing, deciding by the fingerprint fields of revision 86; `stash` and `commit` are also acceptable wherever nothing collides and `aside` wherever there is work to keep (ADR-0023). Its eight phrasings name no resolution. `bench/gold/keep_uncommitted_work_and_sync.yaml` holds a gold program per resolution, each with preconditions written independently of the rules. `tests/test_dirty_tree_intent.py` replays every gold body, conditions stripped, on every injected state: the checker's verdict equals membership in the declared set in all nine cases; each gold program's preconditions fire on its own state only; each resolves its own state with its conditions in place. **It is not registered**: `INTENTS` and `STATE_GRID` do not list it (`DIRTY_TREE_STATES` is declared beside them), the fault stays in `EXCLUDED_FROM_BENCHMARK` and `variant_for_seed` stays `None`, so nothing measured reads it. No measurement changes. |
| 89 | 2026-10-04 | **The trusted pre-fetch follows a default branch upstream renamed (fix to revision 79).** A clone records `refs/remotes/<remote>/HEAD` once, and no fetch updates an existing one -- git >= 2.48 only creates it when missing -- so after upstream renamed `main` to `trunk`, `repository_bindings` still derived `main`, a branch upstream no longer has, and `dispatch` and `learn` would bind every `{upstream_branch}` to it. `open_checkout`'s pre-fetch now also runs `git remote set-head <remote> --auto`, with the user's own git like the fetch, before deriving the parameters; a remote that reports no HEAD leaves the recorded one. Reproduced first: the new test bound `main` before the change and binds `trunk` after it. Stale remote-tracking refs are not pruned; whether the pre-fetch should prune is #190's question. No measurement changes. |
| 90 | 2026-10-04 | **The dirty-tree intent is registered, so `dirty_tree` is measured (issue #189, ADR-0027).** `keep_uncommitted_work_and_sync` joins `tasks.registry.INTENTS`, `dirty_tree` leaves `EXCLUDED_FROM_BENCHMARK` (now `branch_renamed` and `lockfile_conflict`), and its four declared states join `STATE_GRID`. `StateFingerprint.as_text` now renders `dirty_files` and `untracked_upstream_collisions` for every state, so **every arm-2 text in every family gains two lines**, and arm-2 numbers recorded before this revision are not comparable with later runs; the recorded baselines are history and are not recomputed. The fault draws two Tier 1 axes (ADR-0005): `content` (four flavours of every text the injector writes) and `untracked` (whether a non-colliding untracked file accompanies `disjoint` and `same_file`); `aside` varies on `content` only. Measured from the declared draw: the eval set builds 14 `dirty_tree` environments (`aside` 4, `commit` 4, `stash` 6), so the 40 eval seeds build 61 environments across the three faults, and the smoke set draws no `stash` state. **Live corrections:** §3's intents and AUC tables and naming fractions, §7's demo arithmetic (36 episodes, not 24), occurrence and instance tables and interval widths, §10's testing block and open risk 6 now count three measured faults and two excluded ones. `bench.live.MEASURED_FAULTS` is derived from the registry. No eval episode has been run on the new grid, and no result is claimed. |
| 91 | 2026-10-04 | **The fingerprint observes the branch wiring a rename leaves stale, without rendering it (issue #190, step 1).** `StateFingerprint` gains `tracked_branch` (`branch.<branch>.merge`, without `refs/heads/`), `upstream_default_branch` (what `refs/remotes/<upstream remote>/HEAD` names, without the remote) and `local_branches`. The default is read from the recorded remote `HEAD`, which a plain fetch never moves and the trusted pre-fetch brings current with `remote set-head --auto` (#205), so the owner's decision not to prune (#190) costs nothing: the stale old-name ref stays, and `HEAD` says what upstream uses now, read locally with no fetch and no `ls-remote`. `as_text` leaves the three fields out (`signatures._NOT_YET_RENDERED`) until the intent is registered, as revision 86 did for the dirty-tree fields, so **every arm-2 text, and every measured number, is unchanged**. No result is claimed. |
| 92 | 2026-10-04 | **`branch_renamed` injects three states, as a pre-fetched checkout shows them, and its checker protects local work (issue #190, step 2).** Upstream now renames its default branch *and* moves on with one commit there, and each seed selects `plain`, `local_work` (a local commit upstream lacks, so the sync is no fast-forward) or `name_taken` (a local branch already has the new name, with a commit of its own). The injector leaves the sandbox as the trusted pre-fetch leaves a checkout -- fetched, `remote set-head --auto` run, **nothing pruned** (owner's decision, #190) -- so the old name's remote-tracking ref is stale and still resolves, which is the quiet wrong fix the checker must refuse. `check` gains two clauses: the `local_work` commit's change must still be in the tree (graded by presence, as `diverged` does), and the `name_taken` branch's commit must still be reachable from a local branch, so `reset --hard` onto the new name and `branch -M` over the taken name both fail. **Correction to revision 28:** `branch_renamed`'s change surface is no longer empty: a fast-forward or merge brings upstream's `app.py` commit, and the local commit's `docs/readme.md`, into the committed diff from the base, so it declares those two; the empty-surface rule is pinned on its own. `sandbox.harness_upstream_branch` makes a harness sandbox read upstream's default from a recorded `refs/remotes/upstream/HEAD`, as a checkout does (`repository_bindings`), so `upstream_ref` and `{upstream_branch}` agree with a checkout of the same state; no other fault records one, so every other state still reads `upstream/main`. `variant_for_seed` stays `None`, so the fault stays excluded and admission still builds one state of it, at seed 0. No measured number changes and no result is claimed. |
| 93 | 2026-10-04 | **The renamed-branch intent is defined, and its acceptable sets are pinned to the checker -- not yet registered (issue #190, step 3).** `tasks.faults.branch_renamed.INTENT` (`follow_renamed_upstream_branch`) labels `plain` `rename`, `local_work` `merge` and `name_taken` `retrack`, decided by `tracked_branch`, `upstream_default_branch`, `local_branches` and `upstream_behind`; no rule reads the request. Measured by replaying every gold body on every state with its conditions stripped, the checker accepts {merge, rename, retrack} on `plain`, {merge} on `local_work` and {merge, retrack} on `name_taken`, and the declared sets equal that (`tests/test_branch_renamed_intent.py`). This is `diverged`'s shape: `merge` is right everywhere, and the wrong fires are an in-place retrack or rename over local work and a rename onto a taken name. Gold bodies chain their steps with `&&`, so a failed `git branch -m` stops the body rather than letting the rest act as a different resolution. `StateFingerprint.follows_a_renamed_branch` names the situation, and **`diverged`'s rules now refuse it**: without the guard they labelled the renamed `local_work` state `rebase`, a resolution that leaves the wiring stale and passes no checker there. No diverged state records an upstream default, so no label this family has had changes. The intent is not registered and `variant_for_seed` stays `None`, so no measured number changes and no result is claimed. |
| 94 | 2026-10-04 | **The renamed-branch intent is registered, so `branch_renamed` is measured (issue #190, ADR-0028).** `follow_renamed_upstream_branch` joins `tasks.registry.INTENTS`, `EXCLUDED_FROM_BENCHMARK` names only `lockfile_conflict`, and the four declared renamed states join `STATE_GRID`. `as_text` now renders `tracked_branch`, `upstream_default_branch` and `local_branches`, so **every arm-2 text gains three lines** and arm-2 numbers recorded before this revision are not comparable with later runs. The fault draws Tier 1 axes `name` (the new branch name, already bound as `{upstream_branch}`) and `content` (four flavours of each side's texts) for every resolution. `diverged`'s three gold programs gain a precondition that the branch follows the branch upstream uses now: it holds on every diverged state, and without it gold `rebase` fired on the renamed `local_work` state. Measured from the declared draw: the eval set builds 30 `branch_renamed` environments (`merge` 9, `rename` 12, `retrack` 9), 91 across the four faults; occurrence roles are smoke 4/0, tune 15/1, eval 30/10; the text control's uninformed AUC is 0.500 by construction and informed 1.000; 0.135 of uninformed requests over seeds 0–199 name the fault. **Live corrections:** §3's intents and AUC tables and naming fractions, §7's demo arithmetic (48 episodes), occurrence and instance tables and interval widths (Wilson ±10.1pp and TOST ±12.0pp at 91), §10's testing block and open risk 6 now count four measured faults and one excluded. No eval episode has been run on the new grid, and no result is claimed. |
| 95 | 2026-10-04 | **The fingerprint observes files a sync would conflict in, without rendering them (issue #191, step 1).** `StateFingerprint` gains `merge_conflicted_files` -- files both sides changed whose three-way merge conflicts, tried with `git merge-file -p` on the merge base's, HEAD's and upstream's blobs copied to a temporary directory outside the repository, so observing writes nothing, not even an object -- and, in those files, `upstream_dropped_lines` and `local_dropped_lines`: base lines one side removed while the other kept them. A line both sides changed, such as a version bump, is dropped by neither, so only real removals count. `diverged`'s overlapping state, where both sides changed `app.py` in separate hunks, is not a conflict. `as_text` leaves the three fields out (`signatures._NOT_YET_RENDERED`) until the intent is registered, so **every arm-2 text, and every measured number, is unchanged**. No result is claimed. |
| 96 | 2026-10-04 | **`lockfile_conflict` injects three states, and its checker refuses resurrected dependencies and a doubled version line (issue #191, step 2).** The base lock now lists a removable dependency, and each seed selects `additions_only` (each side adds one entry and bumps the version), `upstream_removed` (upstream also drops the removable entry the local side kept) or `local_removed` (the mirror). `check` gains two clauses: a dependency one side removed must stay removed, and the file must have exactly one `version` line -- a union merge driver keeps both sides' lines, so both versions, a lock no real ecosystem could read, which the three-clause checker passed. Measured on real sandboxes (`tests/test_lockfile_states.py`): rebuilding from upstream's file plus the local side's additions passes `additions_only` and `upstream_removed`; rebuilding from the local file plus upstream's additions passes `additions_only` and `local_removed`; a union merge passes none. `variant_for_seed` stays `None`, so the fault stays excluded and admission still builds one state of it, at seed 0. No measured number changes and no result is claimed. |
| 97 | 2026-10-04 | **The lock-conflict intent is defined, and its acceptable sets are pinned to the checker -- not yet registered (issue #191, step 3).** `tasks.faults.lockfile_conflict.INTENT` (`sync_through_a_conflicting_lockfile`) labels `additions_only` and `upstream_removed` `take_upstream` and `local_removed` `keep_local`, decided by `merge_conflicted_files`, the dropped-line fields and both sides having commits; no rule reads the request. Measured by replaying both gold bodies on every state with their conditions stripped, the checker accepts {keep_local, take_upstream} on `additions_only`, {take_upstream} on `upstream_removed` and {keep_local} on `local_removed`, and the declared sets equal that (`tests/test_lockfile_intent.py`). It is its own intent rather than more `sync_fork_with_upstream` states, settling #191's first question: every `diverged` resolution fails here, starting with a plain merge, and **`diverged`'s rules now refuse a state where a sync would conflict** (`StateFingerprint.sync_would_conflict`) -- #158 found them labelling this fault's state `merge`. No diverged state conflicts, so no label it has had changes. Gold bodies pass the other side's additions through a file under `.git/` and insert them with a `\\%...%`-addressed `sed`: the guard refuses a `while read` loop's variable and reads a `/`-addressed `sed -i` script as an absolute path, two limits recorded for the pre-live-run review rather than changed here. **One admission behaviour changes, and it narrows ADR-0019:** a `diverged` program whose preconditions accept the lock-conflict state is now refused at admission as firing on an unrelated state, where #158 and ADR-0019 had judged that state an overlap on which `merge` was correct -- the experiment shows a plain merge stops on the conflict and fails the fault's checker there (`tests/test_overlap_states.py`). The gold merge program's `local_work_beyond_the_overlap` precondition keeps it off the state, so gold is still admitted, and ADR-0019's overlap class stays in `admit` with no measured fault producing one today. The intent itself is not registered and `variant_for_seed` stays `None`. No eval episode has been run, and no result is claimed. |
| 98 | 2026-10-04 | **The lock-conflict intent is registered, so every fault is measured (issue #191, ADR-0029).** `sync_through_a_conflicting_lockfile` joins `tasks.registry.INTENTS`, `EXCLUDED_FROM_BENCHMARK` is empty, and the four declared lock states join `STATE_GRID`. `as_text` now renders `merge_conflicted_files`, `upstream_dropped_lines` and `local_dropped_lines`, so **every arm-2 text gains three lines** and arm-2 numbers recorded before this revision are not comparable with later runs. The fault's Tier 1 axes are each side's package plus `removal`, which separates the two states `take_upstream` covers; the versions now follow from the package pair instead of being drawn on their own, because drawn independently no tune or eval seed ever repeated an instance and the cost curve is read from repeats. Measured from the declared draw: the eval set builds 28 `lockfile_conflict` environments (`take_upstream` 17, `keep_local` 11) with 12 replays, tune 14 with 2, smoke 3 with 1; 119 environments across the five faults; the text control's uninformed AUC is 0.500 by construction and informed 1.000; 0.000 of uninformed requests over seeds 0–199 name the fault. **Live corrections:** §3's intents and AUC tables and naming fractions, §7's demo arithmetic (the nominal 60 episodes is now the runnable grid), occurrence and instance tables and interval widths (Wilson ±8.8pp and TOST ±10.5pp at 119), §10's testing block, and **open risk 6, which is closed**. No eval episode has been run on the new grid, and no result is claimed. |
| 99 | 2026-10-04 | **Git's `@{...}` revision syntax is no longer read as a program placeholder (bug fix, #214).** `runtime.probes._PLACEHOLDER` skipped `${name}` as the shell's but not `@{name}`, so a probe or body using `@{u}`, `@{upstream}` or `HEAD@{1}` was refused as naming an unknown placeholder. The fourth live smoke run found it: a compiled `branch_renamed` program's probe used `git rev-parse --abbrev-ref @{u}` and was refused on the positive side for that reason alone. **This changes admission**: a program using that syntax is now judged on its merits instead of refused, so a build can admit programs it refused before; no other substitution changes (`tests/test_runtime_replay.py`). No eval episode has been run with the fix, and no result is claimed. |
| 100 | 2026-10-04 | **The agent's git tool no longer lets git run a command for it (security fix).** The tool runs one git command with no shell, but git runs commands itself, and the fourth live smoke run's agent used that: with no way to write a file it ran `git config alias.lockfix "!sed -i ... deps.lock"` and then `git lockfix`, so `sed` ran through a shell on the host. `agents.command_screen` now refuses, before either path runs a command: a config write to any key outside an allowlist of branch, remote, submodule, user and behaviour keys, or to a `!` value; an editor; a section rename; `submodule foreach`, `bisect run`, `filter-branch`, `difftool`, `mergetool` and the servers; and the options that name a program (`rebase --exec`, `--upload-pack`, `--receive-pack`, `grep -O`, a template, a clone's config, `--exec-path`). Reads are not limited, and every config command the live agents ran to inspect or repair a repository still runs. **This changes arms 1 and 1b and every fallback**: a solve that reached for one of these forms gets a refusal it can read. The fourth run has one such solve (a lockfile fallback that failed anyway). It is a screen over the argument list, not a boundary: the harness sandbox still has no filesystem isolation (#10). |
| 101 | 2026-10-04 | **The agent can read and write files in the working tree (ADR-0031).** `agents.file_tools` adds `read_file` and `write_file` beside `run_git` and `finish`, confined to `env.work`: a relative path, no `..`, no `.git` component, no symbolic link on the way, 64 KiB each way. The fourth live smoke run solved no lock-file conflict honestly: the resolution is a rebuilt file, the only tool ran git, and the one success wrote its blob through `git notes`. A compiled body could always write a file with shell, so the baseline had less than the programs it is measured against. The compile prompt now says a body must compute a write's content from the repository rather than paste the agent's text. **This changes every solve** (arms 1 and 1b and every fallback): two more tools and a longer system prompt, so arm-1 and fallback numbers are not comparable across it. No eval episode has been run with it, and no result is claimed. |
| 102 | 2026-10-04 | **A predicate's `expect_pattern` binds its placeholders and reads POSIX classes (bug fix).** `runtime.probes.pattern_for` now substitutes `{placeholders}` in a pattern as in the probe, with each value regex-escaped, and turns `[:space:]`, `[:digit:]` and the other POSIX classes into their characters before `re.search`. The fourth live smoke run had two compiled programs refused by the matcher rather than by their behaviour: a renamed-branch postcondition `^refs/heads/{upstream_branch}$` was matched with the braces literal, so a correct retrack failed it, and four `wip_present` checks used `[^[:space:]]`, which Python's `re` reads as a set followed by a literal `]` and never matched. The compile prompt now says a pattern is a Python regular expression with both readings. **This changes admission**: a program using either form is judged on its behaviour instead of refused. No eval episode has been run with it, and no result is claimed. |
| 103 | 2026-10-04 | **The guard no longer refuses two ordinary shell forms (#214, owner decision 2).** A variable assigned by `read` after a compound keyword or its own assignments (`while read -r x`, `\| while IFS= read -r x`, `if read -r x`) now counts as the body's own, so `$x` after it is not an environment read; a read before it, and every unassigned name, is still refused. `sed -i` is now screened by the files it rewrites: the script (the `-e` value, or the first operand) is no longer read as a path, so `sed -i '/^packages/r extra' deps.lock` is allowed, while a script's `w`/`W` file is still a write target and a file outside the sandbox is still refused. A redirect to `/dev/null` inside `$( )` is no longer read as a write to `/dev/null)`, which refused one of the fourth live run's `branch_renamed` probes. **This changes admission**: a compiled body or probe using any of these forms is judged on its behaviour instead of refused. No eval episode has been run with it, and no result is claimed. |
| 104 | 2026-10-04 | **A refused compile gets one revision, told why (ADR-0030).** `bench.run._learn_from_solution` now compiles at most `COMPILE_ATTEMPTS = 2` times: when admission refuses a program, `compile_program(revision=(program, reason))` puts the refused program and admission's reason into the payload (inside the untrusted block, since the reason quotes the model's own program) and asks once more; the revision is judged by the same, unchanged `admit`. The fourth live smoke run's build admitted 4 of 20, with every `diverged` program refused for firing on a renamed-branch state -- correct refusals that no compile was ever told about. **This changes the build and the online arms:** an admission rate now means admitted within one revision, every attempt's tokens are the episode's, and the ledger's new `compile_attempts` separates first-attempt from revised admissions, which the next report must give separately. The build's positive-only library (ADR-0010) is gated on each episode's **first** compile: the revision answers the two-sided gate's refusal, so it stays in the two-sided library, and the refused first program is kept in the stored program's `history.jsonl` (`Library.record_revision`, `first_compile`). When the refusal is a negative state the program fired on, the revision payload also carries that state's observed fingerprint as `refused_state`, so the compiler can see what tells it apart from its own state. No eval episode has been run with it, and no result is claimed. |
| 105 | 2026-10-04 | **An assignment in a `case` arm, or after `then`, `do` or `else`, is the body's own (#214).** The guard's statement start now includes the `)` that ends a case arm's pattern and skips the compound keywords, so `*.*) name=${base%.*}` and `then n=1` assign their names and a later `$name` or `$n` is not an environment read. The fifth live run's compiled `dirty_tree` body was refused for `$name` on its first attempt, which spent its one revision on the guard. A read before the assignment, and every unassigned name, is still refused. **This changes admission** in the same way as revision 103. |
| 106 | 2026-10-04 | **A program refused on the state it was compiled from is told its variant is wrong (ADR-0030).** When a same-intent state the preconditions accept resolves to the same resolution as the program's own positive seed, admission's reason now says that this is the state the program was compiled from and that the declared `variant`, not the preconditions, is what is wrong. The fifth live run compiled an `aside` body from the aside state, labelled it `stash`, and the refusal read like a precondition defect, which is what the one revision would then try to fix. The verdict is unchanged; only the reason a revision reads is. |
| 107 | 2026-10-05 | **A refusal names every state the program wrongly fired on, and the revision sees all of them (ADR-0030).** Admission's unrelated, overlap and same-intent classes now collect every state the preconditions wrongly accept before refusing, instead of returning at the first. The reason gives the first in full and lists the rest; `admit(fired_on=...)` reports each, and the revision payload carries all their fingerprints as `refused_states`. In the sixth live run three `diverged` revisions fixed the renamed-branch state they were told about and were then refused on the lock-conflict state they had not been. The verdict is unchanged: a program refused before is refused now. Admission now builds every negative sandbox for a refused program, which costs time, not model calls. |
| 108 | 2026-10-05 | **A compile reply that does not parse gets the one retry, as a repair (ADR-0030).** `_learn_from_solution` retries a first reply that is not a valid program, with the reply (capped at 8,000 characters) and why it did not parse in the payload; a repair that fails too is named on the row beside the first failure. It shares the budget of two compiles with the revision. The fourth and sixth live runs each lost a build program to an unparsable first reply. **This changes the build and the online arms** as revision 104 did. |
| 109 | 2026-10-05 | **The admit set covers every resolution, and asks for it (ADR-0032).** `bench.admit_set.ADMIT_SET` replaces `SMOKE_SEEDS` with each fault's own request as the build's admit set: one episode per declared resolution of each measured intent, 14 in all, at a seed whose state is labelled with that resolution and, where some state accepts it alone, is such a state; each request is the fault's own phrasing followed by a directive asking for that resolution. Arm 2's floor (item 11) needs two resolutions per intent and no live run met it: given requests naming none, the agent chose `rebase` in all eight `diverged` build episodes and `stash` in every `dirty_tree` one, because most states accept several resolutions and no `diverged` state forces one. The directives are build-only: no measured request, phrasing or arm changes. **This changes the build**: its numbers are not comparable with runs 1-6. No eval episode has been run with it, and no result is claimed. |
| 110 | 2026-10-05 | **The compile prompt says a probe must only read.** Probes have run between two snapshots and been refused when they change the sandbox since issue #10, but the prompt never said so, and its example shows `git fetch` (in a body). The seventh live run's first build program fetched with `--prune` in a precondition, pruning a remote-tracking ref, and was refused as unevaluable. The prompt now states the rule and that fetching is the body's job. **This changes the compile prompt only**; admission is unchanged. |
| 111 | 2026-10-05 | **The compile prompt says how to check that something is absent.** A predicate holds when the probe's exit code matches and, if a pattern is set, the pattern is found; nothing expresses "the pattern is absent". The eighth live run's `dirty_tree` `stash` program wrote its "stash entry consumed" postcondition as `git stash list` expecting exit 1 and its stash message as the pattern, which can never hold, and was refused on the positive side through its revision. The prompt now says a pattern only requires presence, to write an absence check as a probe that exits 0 only then, and that many git commands exit 0 whatever they print. **This changes the compile prompt only**; admission is unchanged. |
| 112 | 2026-10-05 | **The agent is told its tool-call budget, and reminded with eight calls left.** `react.system_prompt(max_tool_calls)` states the budget and asks the agent to investigate only what the request needs and leave room to finish; `NUDGE_TOOL_CALL_MARGIN` is 8, from 4. The eighth live run's `dirty_tree` aside solve spent 20 of its 24 calls exploring (reflog, config, tags), was reminded with four left, wrote its two `.local` files, and had its `git clean`, `git restore` and merge refused; earlier lockfile solves ended the same way. **This changes every solve** -- arms 1 and 1b and every fallback, identically -- so arm-1 and fallback numbers are not comparable across it. No eval episode has been run with it. |
| 113 | 2026-10-05 | **Git's `^{type}` peel syntax is no longer read as a program placeholder (bug fix).** `runtime.probes._PLACEHOLDER` skipped `${name}` and `@{name}` but not `^{name}`, so a probe using `rev^{commit}` or `^{tree}` was refused as naming an unknown placeholder. The eighth live run's `diverged` discard revision -- shown, by revision 107, every state its first program wrongly fired on -- used `{upstream_remote}/{upstream_branch}^{commit}` and was refused for it. **This changes admission** in the same way as revision 99. |
| 114 | 2026-10-05 | **Each admit-set resolution has a fallback seed (ADR-0032 decision 5).** `AdmitEpisode.seeds` lists a first seed and a second at the same kind of state; the build runs the second only when the first episode admitted no program, and stops a resolution at its first admission. In the eighth live run one failure per resolution was enough to leave `dirty_tree` and `diverged` with one resolution each, and the floor unmeasurable again. The build runs at most 28 episodes; each resolution is still covered by at most one admitted program from its own episodes. **This changes the build** as revision 109 did. |
| 115 | 2026-10-05 | **Admission builds every injected state, not one per label (bug fix).** `_state_seeds` keyed states on `variant_for_seed`, the label, so two states sharing a label were one: lockfile's `additions_only` and `upstream_removed` both label `take_upstream`, and only `additions_only` was ever built. `keep_local` is acceptable there and wrong in `upstream_removed`, so a `keep_local` program firing on every lockfile state passed every negative class. The eighth live run admitted one, and it was all sixteen of arm 3's mismatches on the 400 eval pairs. `FaultSpec.state_for_seed` now names the state, and the negative classes and the breadth universe (now 16 states) enumerate states with it. **This changes admission**: such a program is now refused. |
| 116 | 2026-10-05 | **A refused account stops the run instead of failing its episodes.** HTTP 401, 402 or 403 raises `provider.ProviderAuthError`, which `solve`, its last word and the compile step re-raise rather than record; `bench.live` exits with "the provider refused the account" and leaves no summary. The tenth live run's revoked key otherwise wrote 28 failed build rows and a summary in under a minute -- a run that looked measured and measured nothing. Every other provider error is still recorded on its episode, and the retry policy is unchanged. No measured number changes: a run that meets a refusal now has no rows rather than wrong ones. |
| 117 | 2026-10-05 | **A live run interrupted during its build resumes.** `bench.live --resume` (and `run_live(resume=True)`, `build_library(resume=True)`) continues the run in `--out`: each finished build episode is committed to `build.progress.jsonl` once its programs are in both roots, a resumed build skips every committed one and re-reads its programs from there, and the programs an interrupted episode had copied are removed first. `plan.json`, written first, holds the run to the same plan and model; a run whose summary is written, or that got past its build, is refused. The fifth and ninth live runs lost their builds to container restarts at 12 of 20 and 14 of 28 episodes. **No measured number changes:** a resumed build runs the same episodes in the same order, and an episode cut off mid-way runs again in full -- its spend before the cut is not on any row, and its row stays if it was written. |
| 118 | 2026-10-05 | **A compiled reply's YAML is read as written, or refused with why.** PyYAML reads a plain `probe: ! cmd` as the tag `!` on `cmd` and drops the `!`, so the absence check the compile prompt's own rule shows (#234) loaded inverted. `agents.compile._parse_document` now refuses any YAML tag, anchor or alias, and every refusal -- those, a YAML error with its line, a non-mapping top level -- is appended to the row's reason and so reaches the repair retry, which until now was told only "not a YAML mapping". The prompt says which shell starts YAML misreads and asks for a block scalar (`probe: \|-`). The tenth live run lost three of its first seven compile replies to unparsed YAML and was stopped after four episodes; its replicates restart on this. **This changes every compile** -- identically across arms -- so admission rates are not comparable across it. |
| 119 | 2026-10-05 | **awk's braces are not program placeholders.** `runtime.probes` leaves a `{word}` as written when it cannot be a parameter: outside `VOCABULARY`, with no underscore, and not itself a word of a parameter. The eleventh live run's compiled lockfile probe `awk '/^--- / {next} /^-/ {found=1} END {exit !found}'` was refused for naming an unknown placeholder `next`. A misspelt parameter (`{upstream_brnch}`) or one reached for by a word of its name (`{branch}`, `{upstream}`) still raises, and `placeholders` -- admission's declaration check -- reads names the same way. The eleventh to thirteenth runs (the three replicates) were built before this, so their libraries can hold this false refusal; later runs are not comparable on lockfile admission. |
| 120 | 2026-10-05 | **A placeholder after a regex anchor is substituted.** #236 exempted every `^{name}` as git's peel syntax, so `expect_pattern: ^{upstream_branch}$` -- the commonest anchored pattern -- stayed literal and could never match. `runtime.probes` now leaves `^{name}` to git only for its peel types (`commit`, `tree`, `blob`, `tag`, `object`). The twelfth live run (replicate 2) refused a correct `rename` program for it on its first episode and was stopped; the eleventh (replicate 1) was built with the defect, as were the eighth onwards, so their admission counts are not comparable with later runs. |
| 121 | 2026-10-05 | **A Wilson interval always contains its point, and a run interrupted in its pair-level stage resumes.** `bench.report.wilson_interval` bounds the interval by its point: at 0 and 1 floating point left a bound a rounding error on the wrong side (0/11's low above 0, 6/6's high below 1; 90 such (k, n) with n < 200). The eleventh live run's Figure 1 plot refused the negative error bar that made and crashed after its build, losing its summary; `primary._errors` also clamps at 0. `bench.live --resume` now accepts a run interrupted in the model-free pair-level stages and recomputes them whole from the finished library; an episode stage is still never resumed. Intervals move by at most a rounding error. |
| 122 | 2026-10-05 | **A finished run's primary metric can be re-measured with another arm 2 scorer.** `bench.rescore --run DIR --scorer {lexical,embedding} --out DIR` opens the run's frozen two-sided library with the chosen scorer behind `Library(similarity=...)` and runs `bench.live`'s own pair-level stage -- 2b threshold, arm 2's floor, Figure 1, Claim 2 -- into a fresh directory, recording the scorer in `rescore.json`. No model call and no rebuild: arm 3 does not read the seam, so only arm 2's curve, floor and matched comparison move. It is how #104's embedding scorer is measured against the three replicate libraries; `lexical` reproduces a run's own figures. |
| 123 | 2026-10-10 | **Pre-registration revision: arm 2 gains a judge that reads the state, and its first run is gated on its own floor (issue #264, ADR-0034).** §7's pre-registered analysis gains item 14. The judge is `bench.judge_similarity.JudgeSimilarity` behind the `Similarity` / `ScoresMany` / `ReportsUsage` seam, so nothing downstream of the seam learns what it is and **arm 3 is unchanged by construction**. It sees exactly what arm 2 sees -- the request, `StateFingerprint.as_text()`, and each candidate's `_program_text` -- and **not** the probe strings, because a judge that reads the probes is reasoning about the predicates arm 3 executes (a separate experiment, in ADR-0034's rejected table). It makes **one provider call per pair** (`score_many`) returning one score per candidate, so item 1's sweep still applies rather than degenerating into a step; model, temperature and prompt are pinned in code and their hash travels with every artifact; its per-pair scores are **committed as a JSONL cache**, so a figure can be recomputed with **no key and no tokens**, and a cache whose prompt hash, model or per-record candidate count disagrees is refused rather than reused. Failed replies -- empty completion, transport error, unparseable, wrong length, out of range -- become an **abstain** (`0.0` for every candidate) and stay in item 1's denominator, counted separately; `ProviderAuthError` is re-raised and stops the run (issue #157's precedent) and there are no silent retries. **The floor comes first:** the judge's informed-regime strict top-1 on the tune seeds against each frozen library is judged by item 11, and below it no "beats text similarity" sentence may be written. **Repeatability is registered, not assumed:** a pre-registered subsample is asked `k >= 2` times and the agreement is reported beside the judge's row, with the statement that the three text scorers carry no such component -- a judge is an LLM, and the primary metric moved to pair level precisely because sampling variance made the episode-level comparison unreadable (revision 41 records that temperature 0 is not determinism on a hosted model). Its spend is metered in `SimilarityUsage`'s own currency and recorded in the rescore's record, never folded into the LLM's `tokens_in`/`tokens_out` nor into an episode's `embedding_*`. **Registered before the first judge call:** no judge has been called, the floor and the three rescores need a provider key the caller supplies, and this revision reports no number. No existing metric definition, denominator or committed figure changes. |
| 124 | 2026-10-10 | **Pre-registration revision: a hybrid dispatcher is measured as a family, and read at arm 3's coverage (issue #265, ADR-0035).** §7's pre-registered analysis gains item 15. The hybrid is four steps -- rank the admitted programs by the text scorer unthresholded, keep those at or above a retrieval floor within the top `k`, fire the best-ranked one whose preconditions all hold, else fire nothing -- and it is **derived from the pair pass's record**: `bench.library_pairs` now records each environment's accepted program ids and each request's complete text ranking, both of which it was already computing and discarding, so the whole grid is arithmetic over one pass rather than a sandbox per point. The registered grid is `k` in `(1, 2, 3, 5, 10, None)` crossed with every distinct retrieval score plus no floor (item 1's rule). **The claimable comparison is the point nearest arm 3's coverage**, a tie to the lower coverage, with its Wilson interval, the coverage gap and the smallest detectable difference at its fire counts; every other point is the family. **Selection is registered as this issue's trap:** with two knobs over a few hundred points on 400 pairs the best cell is a maximum, so the widest zero-mismatch point per `k` is reported as a description of these pairs, bounded by the 95% Wilson upper limit of the mis-fires it did not have, and never as an unbiased estimate of the rate at a floor chosen because it looked good. **What would falsify it:** at arm 3's coverage the hybrid mis-fires more than arm 3, or no point above arm 3's coverage is reachable without a rise distinguishable at the registered difference. A named operating point's `k` and floor come from the tune seeds, and the stage reads a committed frozen library read-only (digest compared before and after) with no model call and no key. No existing metric definition, denominator or committed figure changes. |
| 125 | 2026-10-10 | **Pre-registration revision: the primary metric gets a held-out state source, and the stage that reads it may not word a win (issue #263, ADR-0033).** The claim's states came from the injectors admission also tested against, so `0 of 563` is an in-distribution figure. §7's pre-registered analysis gains item 13: a plan block `bench.splits.HELD_OUT_SEEDS` of **40 seeds at 3000-3039** -- the eval block's size, chosen against the measured cost of one held-out environment (a real sandbox plus every admitted program's probes) -- disjoint from the smoke, tune, eval and admit sets and from admission's own seed search; `dirty_tree`'s injector gains a **path axis** drawn only above `tasks.spec.HELD_OUT_SEED_BASE`, so no shipped seed's state moves and every committed figure stays recomputable; `bench.heldout` measures the block against a **committed** frozen library read-only (digest compared before and after, no model call, no rebuild) and reports Figure 1's metric with the power statement. **The reading is at the instance, not the seed or the pair:** the 40 seeds build 31 distinct instances (`FaultSpec.instance_for_seed`, reported as `distinct_instances`), and since arm 3 is request-blind the requests crossed with one environment are one decision rather than several observations (ADR-0005 decision 4). Because item 11's floor is measured on the tune seeds against the shipped states, it does not transfer, so the stage carries a `WORDING_LIMIT` field and no "beats text similarity" sentence may be read off it. Where arm 3 decides every held-out pair correctly the comparison is vacuous, which on **compiled** programs is a result rather than a tautology: the interval on its rate is what may be read, and the arm-2 contrast waits for a block on which arm 3 abstains somewhere. **Logged before the held-out block was measured, and the one thing measured before it is disclosed:** two seeds were built first to verify the stage runs end to end -- 4 pairs, arm 3 firing 2 with 0 wrong, comparison vacuous -- and **no analysis is read from them**: they are excluded from every figure, and the reading rule decision 5 states was written after that smoke showed a compiled arm 3 can be perfect on the block. The block's own seeds had not been built when this row was written. No metric definition, denominator or reported number in this document changes, and the three replicate figures stand as in-distribution. |
| 126 | 2026-10-10 | **A git path list is read one path per line, because a path may contain a space (issue #263).** The held-out path axis failed on its first build: seed 3006 draws a `same_file` state whose upstream file is `docs/release notes.md`, and `StateFingerprint.observe` read `git diff --name-only` with `str.split()`, so one path became two entries -- `upstream_touched_files` held `['docs/release', 'notes.md']` while `dirty_files`, which already used `splitlines()`, held the path whole. The two lists then disagreed about the same file, the state read as `disjoint`, and `build_sandbox` refused the build as mislabelled (ADR-0005 decision 3) -- the invariant doing its job, on a defect in the fingerprint rather than in the injector. `local_touched_files`, `upstream_touched_files` and `tasks.invariants.recorded_state_intact`'s change-surface diff now use `splitlines()`, as the other field readers already did. **No shipped fingerprint moves:** every shipped path is space-free, so the two readings agree on every state the three replicate libraries were built and measured from, and no metric definition, denominator or reported number changes. The refused state is the regression test (`tests/test_heldout_states.py`). Not fixed, and not on this stage's path: `bench.instance_diversity._submodule_paths` reads a `.gitmodules` path with `split()[-1]`, so a submodule path containing a space would be truncated there; no injected submodule path has one. |
| 127 | 2026-10-12 | **A program's preconditions must hold on its own positive states (ADR-0036).** Admission's positive side replayed a program on its learning state but never evaluated its preconditions there, so a program whose preconditions never hold passed every side -- its replay works and it refuses every negative -- and was admitted unable to fire. `results/abstentions-2026-10-12/` found one in replicate 2's library, which cost all 13 of that library's arm 3 misses. The positive side now evaluates the preconditions on each fresh positive sandbox before the replay, under both gates, and refuses with the failing predicates. The committed libraries are not rebuilt: a dead program never fires, so their figures stand. |
| 128 | 2026-10-16 | **`diverged` has a held-out shape, and the held-out stage measures it (issue #263, ADR-0037).** Above `HELD_OUT_SEED_BASE` a `diverged` state builds 2-3 upstream commits, an upstream-only file and 4-5 local commits; below it every seed keeps the shipped shape and instance identity. `bench.heldout.HELD_OUT_FAULTS` becomes `("dirty_tree", "diverged")`, and `WrongFire` records its seed. On the three committed libraries arm 3 fired on all 40 held-out `diverged` states with 0 wrong; on `dirty_tree`'s held-out paths it mis-fired once, on replicate 3, where a probe compared quoted `git status --porcelain` paths with unquoted `git diff --name-only` ones (`results/heldout-2026-10-16/`). |
| 129 | 2026-10-17 | **The compile prompt says how git quotes paths (ADR-0038).** Arm 3's first held-out mis-fire came from a probe that compared `git status --porcelain`, which quotes a path with a space, with `git diff --name-only`, which does not. Both escape a non-ASCII name, and `core.quotePath=false` does not stop `status` quoting a space. The prompt now names which commands quote what, says to read every path list with `-z`, and to list untracked files with `git ls-files --others --exclude-standard -z`. `tests/test_path_quoting.py` runs the prompt's own example on spaced and non-ASCII collisions and on held-out states 3038 and 3010. An audit of the committed libraries found 15 probes in 9 of 40 admitted programs that a space or a non-ASCII name could break; 3 were confirmed by running, 12 found by reading. **This changes the compile prompt only**, by 865 characters, identically across arms; admission is unchanged and no committed figure moves. |
| 130 | 2026-10-18 | **`lockfile_conflict` has held-out paths, and the fingerprint reads git paths with `-z` (issue #263, ADR-0039).** Above `HELD_OUT_SEED_BASE` the lock-shaped file moves from `deps.lock` to a nested path, a path with a space, a non-ASCII name or a dotfile; below it every seed keeps `deps.lock` and its identity, the checker grades the recorded path, and `bench.heldout.HELD_OUT_FAULTS` gains the fault. Building the non-ASCII state showed `StateFingerprint.observe` missing the conflict, because `git diff --name-only` escaped the name; every path list is now read NUL-separated. Of 410 shipped fingerprints compared none moved, and six held-out `dirty_tree` ones changed only in how a non-ASCII name prints; those committed held-out results were regenerated with arm 3's counts and arm 2's matched point unchanged. The gold `lockfile_conflict` programs now require `deps.lock` on both sides, which they rebuild by name. On the three committed libraries arm 3 mis-fired on 41 of 101 held-out lock-file fires, all on the spaced or non-ASCII path, from four probes; ADR-0038's audit missed one of them, so its count becomes at least 16 probes in 10 programs (`results/heldout-2026-10-18/`). |
| 131 | 2026-10-18 | **A repository-derived value is bound only if a shell cannot read it as code (ADR-0040).** `substitute` pastes a bound value into probe and body text as written, and both run under `bash -c`. On a checkout (#181) `upstream_branch` is the remote's recorded default branch and `submodule_path` the first `.gitmodules` path, both written by someone other than the user. Measured before the fix: a remote whose default branch is `x;>PWNED` made a probe naming `{upstream_branch}` create `PWNED` in the checkout, and the snapshot refused the probe only after it had run. `bindings` and `repository_bindings` now leave unbound any repository-derived value that is not `shell_safe` (letters, digits, `._/+-`, no leading `/` or `-`, no `..` component), so a program needing it is not applicable; `upstream_ref` then raises `NoUpstreamError` as for any checkout with no derivable upstream. `.gitmodules` is read with `-z` by one function, `sandbox.gitmodules_path`: both readers split the plain output on whitespace, so a submodule at `my lib` bound as `lib`. §9's threat table gains the row. Every harness binding is unchanged, so no committed figure moves. |
| 132 | 2026-10-19 | **A sixth family, `config_renamed`, whose state is config files rather than git history (issue #266 step 1, ADR-0041).** A synthetic app keeps a key-rename table and its settings under `config/`; a seed builds `stale_key` (rename it), `both_keys` (delete the dead old line) or `local_override` (rename it in the git-ignored override, which must stay untracked). The fingerprint gains `config_stale_keys`, `config_keys_set_twice` and `config_ignored_files`, read from files and **rendered only when set**, so every earlier state's text is unchanged: 610 fingerprints compared before and after, none moved. `update_renamed_config_keys` is registered with gold programs that read the rename table; replay pins each state to exactly its own resolution. The guard now recognises a quoted assignment before `read` (`IFS=' =' read -r k v`), which it refused as an environment read. Future plans measure six families (480 pairs; the eval set's 146 environments); committed runs are unaffected, since rescoring reads each run's `plan.json`. Steps 2 and 3 of #266 (a build of the family, and one with a second model) need a provider key. |
| 133 | 2026-10-19 | **`branch_renamed` and `submodule_moved` have held-out axes, so every git fault has one (issue #263, ADR-0042).** Above `HELD_OUT_SEED_BASE` upstream renames its branch to `release/2026`, `main-v2`, `stable.next` or `team/integration`, and the submodule lives at `third_party/libs/core`, `.deps/libcore`, `lib/core-v2` or `ext/libcore`; every value is shell-safe (ADR-0040), every label is kept, and all 3,000 seeds below the base keep their shipped draw and identity. The submodule checker grades the recorded path and its surface is per seed. On the three committed libraries arm 3 fired on all 240 held-out positives, none wrong, and on no fault-free negative (`results/heldout-2026-10-19/`). No compiled program for these faults names a branch or a submodule path; each binds the parameter. Across the five held-out blocks, a value a program receives as a parameter never broke it, and a value it must read from git's printed path lists broke it on 42 of 90 fires with a space or a non-ASCII name. |
| 134 | 2026-10-20 | **A sixth held-out source builds the eval states over real repositories' histories (issue #263, ADR-0043).** Inside `sandbox.real_host_bases` a sandbox's upstream starts from a pinned public repository's commit and the 50 commits before it (`psf/requests`, `pallets/click`, `junegunn/fzf`, `expressjs/express`); the generator's base is committed on top and the fault injected as before. Only each host's URL and commit are recorded; the content is fetched once by commit, under the operator's environment, into a cache that is never committed, and probes still run only in the disposable sandbox, so no program is pointed at a real repository. The seeds are the eval block, paired with Figure 1, and below `HELD_OUT_SEED_BASE`, so no held-out value is mixed in. `submodule_moved` adds its submodule with `--force`, since fzf ignores `vendor`; synthetic states are byte-identical. On the three committed libraries arm 3 fired on 187, 187 and 189 of the 200 positives, none wrong, and on none of the 200 fault-free negatives: the counts Figure 1 measured on the synthetic base (`results/realhost-2026-10-20/`). No floor is measured, so the held-out stage's wording limit applies unchanged. |
| 135 | 2026-10-21 | **An exploratory stage replays each dispatch arm's fires and grades what they leave (ADR-0044).** `bench.consequences` takes Figure 1's own decisions on the eval pairs -- arm 3's first accepted program, arm 2's top-1 at its matched threshold, which reproduces each committed report exactly -- and replays each fired program on a fresh build, with no model call. A fault's state is graded by the episode loop's two clauses (the recorded state over the change surface, then the fault's checker); a fault-free state by whether HEAD, the branch, any ref or the status moved. Arm 3's 563 fires fixed 536; the 27 that did not are run 13's, from two admitted programs that pass their own postconditions while the checker refuses the result, one compiled from a solve the checker had refused. Arm 2's 318 fires on fault-free states changed nothing; 147 of its 241 on fault states fixed them. No replay lost a recorded commit or rewrote upstream. It measures fires only, so it is not the pre-registered episode stage and no registered metric, denominator or wording changes (`results/consequences-2026-10-21/`). |
| 136 | 2026-10-21 | **Four graders are corrected, found by replaying the held-out and real-host fires (ADR-0045).** `recorded_state_intact` reads the committed diff with `-z`, so a non-ASCII surface path is not read as outside it; the `lockfile_conflict` checker matches git's start, end and base markers, not a heading underline; the `dirty_tree` checker requires the work's file to hold the text the injector appended, instead of reverse-applying a patch `git apply` pins to the top of the file; and the gold `aside` walks git's `-z` lists, which the guard now allows by skipping a `read` option's argument (all 605 committed and gold bodies keep their verdicts). Each defect refused a correct resolution; each fix has a test whose negative case is still refused. The eval replays reproduce byte for byte on the fixed code. Over the pinned real histories every replayed fire has the synthetic base's outcome, row for row; on the held-out block arm 3's 539 correct choices fixed 503 and its 42 wrong choices fixed none, every failure a program's (`results/consequences-heldout-2026-10-21/`, `results/consequences-realhost-2026-10-21/`). No registered metric, denominator or figure moves. |
| 137 | 2026-10-21 | **The gold sweep becomes a stage and a CI check (ADR-0045, amended).** `bench.gold_sweep` replays each hand-written gold program where its own preconditions hold and the state accepts its resolution, grades it with the episode loop's two clauses, and exits non-zero on a refusal or on a gold mis-fire. On the eval, held-out and real-host sources it reports none: 240, 200 and 240 graded, with exactly one gold program applicable per state. Held-out `lockfile_conflict` has no applicable gold (the gold requires `deps.lock`), so that grader is not covered there. `tests/test_gold_sweep.py` runs a sample, including a local repository standing in for a real host; with each ADR-0045 fix reverted it catches three of the four, and the fourth (the `-z` diff) is guarded by its own test. No metric, denominator or figure changes. |
| 138 | 2026-10-24 | **The gold `lockfile_conflict` programs find the conflicted file (ADR-0046; supersedes ADR-0039 decision 4).** Each finds the one path a merge would leave conflicted (`git merge-tree --write-tree --name-only -z`, read-only), requires it to be lock-shaped on both sides, and reads it quoted in every probe and the body; the marker postcondition reads git's start and end markers in tracked files. On all 232 shipped (seed, resolution) combinations the old and new programs agree on firing, on the replay and on the tree it leaves. At every held-out path and both resolutions only the state's own program fires, the gold sweep grades all 40 held-out lock-file states with no defect, and the CI sample now catches all four ADR-0045 defects. No metric, denominator or figure changes: gold is read only by the oracle arm, which no committed run used. |

---

## 1. Summary

An LLM agent re-derives the same solution every time it meets the same problem.
This project builds an agent that solves a task once, compiles its solution into
a persistent **program**, and then replays that program on later occurrences
**with zero LLM calls**.

The uncontroversial half is the amortization: cost is paid once and spread
across recurrences. The contestable half — and the reason this is a research
project rather than a caching exercise — is **how the agent decides a stored
program still applies**:

| | mechanism | nature |
| --- | --- | --- |
| arm 2 | similarity between the new task and the one that produced the program | probabilistic |
| arm 3 | the program's own **executable preconditions** accept the environment | checkable |

### Claims

**What this project does not claim.** Added 2026-09-13 after the prior-art sweep;
ADR-0001 records the reasoning and the source-checked citations.

- Not that compiling a task into a persistent program and replaying it cheaply is
  novel — PreAct, SkillDroid, Auto, and LATM all do it.
- Not that dispatching a cached plan by executable preconditions is a novel
  mechanism — it is MACROPS, 1972, with an explicit replan fallback.
- Not that "the most similar case is not the most reusable" is new — Smyth &
  Keane, 1998.

**Claim 1 (context, not a contribution).** Tokens and LLM calls per episode fall
across repeated occurrences for the compiled arms and stay flat for the ReAct
baseline, **read off the cumulative amortized curve with its break-even marked**
(Figure 2) — with the arm triple reported alongside: success rate, tokens per
episode and **cost per success** (§7 item 8), since an arm that succeeds more often
may legitimately spend more. Two limits are part of the claim rather than caveats
around it:

- **It is a claim about tokens and calls, not about currency.** No cost figure is
  reported, because pricing input correctly needs a rate table and DeepSeek's rates
  differ by peak and off-peak hours, so the rate or the call time would have to be
  recorded per episode (§7's ledger bullet). Input is metered as uncached,
  cache-read and cache-write so a cost figure can be computed once rates exist;
  until then the figure is tokens.
- **"Equal success rate" is not available to it.** Comparing the two dispatch arms'
  success rates requires the pre-registered equivalence test (§7 item 10); where it
  does not pass, the text says "comparable success rate". No episode has been run,
  so nothing is equal or comparable yet.

This is an engineering assumption and a cost model. It is established prior art and
is never presented as a finding.

**Claim 2 (primary, empirical form).** At **matched dispatch coverage**,
executable-precondition dispatch is *expected* to mis-fire less often than
text-similarity dispatch — a mis-fire being a program that runs, claims success,
and did not — with mismatch logged independently of episode success. The mechanism is
classical; what has not been measured is the comparison. Whether the expectation
holds is the open question: §7 pre-registers the analysis, §13 lists the works
that bracket it without answering it, and **no result exists yet**. This is the
only *measurement* prior art does not settle.

**Claim 3 (artifact-level).** A negative-sandbox admission criterion — admit a
program only if it passes postconditions on a faulted instance **and** fails its
preconditions on generated negative states — is a requirement that **no confirmed
work publishes**; the closest published gate is exit-condition-based and applied
as a cascade after semantic recall, which is a different test. Cheapness and
enforceability are design intentions, not findings. Reported as a 2×2 factor
against an otherwise identical positive-only gate.

The claim structure changed because the original primary claim was untestable as
designed: with one fixed task sentence per fault, the task text *was* the
ground-truth label, so similarity dispatch could not mis-fire by construction.
Analysing it would have produced a number that looked like a result while
measuring nothing.

### What would falsify this

- **No difference at any coverage.** Then the finding is that executable
  preconditions buy no dispatch accuracy over text similarity in this domain, and
  the negative-sandbox admission factor (Claim 3) becomes the contribution.
- **The injected faults leak a marker.** If faults are recognisable from the task
  text, similarity dispatch wins for the wrong reason. A phrasing that names a
  resolution is rejected directly (`test_no_phrasing_names_a_resolution` matches
  whole-word variant ids in both wording channels), which is the case the AUC gates
  cannot see: the uninformed AUC is fixed at 0.500 by construction whatever the
  words say. A paraphrase that reveals the answer without naming it is a review
  obligation, not a test (§10).
- **Probes encode the answer.** If authoring the precondition vocabulary requires
  the knowledge being measured, the comparison is confounded by author effort
  rather than measured.
- **Compiled programs rot as repositories drift.** Then program mortality, not
  dispatch accuracy, becomes the finding.
- **Amortization never pays.** Compilation costs more than it saves at any
  realistic recurrence count. This no longer threatens a claim — Claim 1 is not
  claimed — it only makes the harness uneconomical.

---

## 2. Goals and non-goals

### Goals

1. A falsifiable, measurable comparison between similarity and
   precondition-gated dispatch, with a baseline an unsympathetic reviewer
   accepts.
2. A working agent that solves branchy git maintenance chores.
3. An experiment affordable for one person in 2–3 weeks, with a pre-registered
   decision rule so the analysis cannot be reverse-engineered from the results.
4. A committed program library whose git history is itself evidence.

### Non-goals

- A product, a UI, or a DSH plugin. (A plugin wrapper is a *possible* phase 6,
  deliberately out of scope — see §12.)
- Beating a frontier model on general agentic tasks.
- Claiming that compiled programs are safe to run against real repositories.
  They are not, and the design exists to keep them off real repositories (§9).
- Local-model support. The provider boundary permits it; nothing implements it.

---

## 3. The task family

### Selection criteria

A fault is admissible only if it satisfies all three. These are load-bearing,
not preferences:

1. **It recurs.** Amortization needs a second occurrence; a one-off task can
   never exercise the amortization cost model, so it cannot inform even the
   secondary concern.
2. **Correctness is checkable without a model.** If scoring an episode requires
   an LLM, scoring costs exactly what the project exists to avoid paying, and
   the cost comparison becomes circular.
3. **It is branchy.** A fault solvable by one aliased command has a *baseline
   cost of zero*, so there is nothing to save. This criterion excludes most
   obvious "agent does devops" tasks and is the reason the family below is
   narrow.

### The faults

| fault | why it recurs | why it is branchy |
| --- | --- | --- |
| `dirty_tree` | syncing a fork with local work in progress is routine | correct recovery depends on whether the edit conflicts with upstream's changes |
| `diverged` | local and upstream both moved | merge vs rebase vs reset depends on whether local commits are worth keeping |
| `branch_renamed` | upstream renames a default branch once, then forks stay broken | requires detecting the disappearance of a remote-tracking branch and re-pointing the local one |
| `submodule_moved` | submodule pins drift constantly in fork-heavy repos | recovery differs by whether the submodule is initialised, dirty, or absent |
| `lockfile_conflict` | dependency bumps conflict in generated files | the right fix is regeneration, not marker-editing — and marker-editing *appears* to succeed |
| `config_renamed` | apps rename config keys, and old names linger in shared and private config files | whether the old name stands alone, beside its new name, or only in a git-ignored override decides rename vs delete vs migrate-in-place; its state is the files, not git history (ADR-0041) |

`lockfile_conflict` is included as much for the **negative sandboxes** it
supplies as for its positive ones: it is the fault most likely to produce a
program that reports success while being wrong, which is precisely the failure
Claim 2 measures.

### Intents and resolutions (revised 2026-09-13)

A fault's request text used to be one fixed sentence, which made the text a
perfect class label and the primary claim untestable. The task family is now
described by **intents**, each carrying a paraphrase distribution and one or more
**resolutions** decided only by observable state:

| intent | resolutions | decided by |
| --- | --- | --- |
| `sync_fork_with_upstream` | discard / rebase / merge | whether local-only commits change files at all, and whether they touch files upstream also changed |
| `restore_submodule_state` | init / repin / remove | whether the submodule is initialised, whether upstream still references it, whether the recorded pin matches |
| `keep_uncommitted_work_and_sync` | stash / commit / aside | whether an untracked file sits where upstream now tracks one, and whether the uncommitted edits touch a file upstream changed (ADR-0027) |
| `follow_renamed_upstream_branch` | rename / retrack / merge | whether the branch follows a name upstream no longer uses, whether local commits exist, and whether a local branch already has the new name (ADR-0028) |
| `sync_through_a_conflicting_lockfile` | take_upstream / keep_local | whether a sync would conflict in a file both sides changed, and which side dropped lines the other kept (ADR-0029) |
| `update_renamed_config_keys` | rename / drop_old / migrate_local | whether a config file still sets a renamed key, whether it sets the new name too, and whether the file is git-ignored (ADR-0041) |

Only intents with two or more resolutions have an experimental surface, and
`tasks/registry.py` is the single place that distinction lives, so an
unambiguous intent cannot be counted into a dispatch rate as if it were evidence.
A resolution is `None` for a *benign* state — nothing to do — and such pairs are
labelled negatives, because a dispatcher that always fires can only be caught by
states that require refusal.

**A resolution is defined by the task family, not inferred from the outcome.**
That matters for one state, and knowing it prevents a misreading of the misfire
rate: on `overlapping_files` the intent requires `merge`, but a `rebase` there also
preserves the work and reaches a synced tree, so the outcome checker cannot tell
them apart — only `discard` is rejected, because only `discard` loses work. The
requirement is the policy itself: shared history is not rewritten when both sides
touched the same file. A dispatcher firing `rebase` on that state is therefore a
misfire **against the label**, and it is scored independently of the episode's
success, which is why the record stores the two as separate facts (§7). Read the
misfire rate as disagreement with the required resolution, not as a count of
destroyed work.

The request text is sampled deterministically from the paraphrase distribution
(`sha256` over the seed, never `hash()`, which CPython salts per process). When the
state is already known, `variant_phrasings` may supply *informed* wording that
reveals the situation to a careful reader, as a real user's description often does;
state may reach the request only through that declared map. The fraction of
sampled uninformed requests that name the fault is documented and asserted:
`sync_fork_with_upstream` 0.000 over seeds 0–49, `restore_submodule_state`
0.060 over 0–49 and 0.125 over 0–199, `keep_uncommitted_work_and_sync` 0.000
over 0–199, `follow_renamed_upstream_branch` 0.160 over 0–49 and 0.135 over
0–199, and `sync_through_a_conflicting_lockfile` 0.000 over 0–199, all at ≤ 0.35
(`tests/test_intent_ambiguity.py`).

#### The boundary condition

The request text has two declared channels, and the primary claim is measured on
only one of them:

- **uninformed** — the shared `phrasings` list: what someone says when they do not
  know what is wrong, or when nothing is wrong.
- **informed** — a resolution's `variant_phrasings` entry: wording that reveals
  the situation.

How far the text alone gets a dispatcher is measured on the informed channel
rather than assumed; on the uninformed channel it is fixed by construction.
`bench/textcontrol.py` trains a bag-of-words logistic regression on the request
text alone and reports AUC per intent and regime, over the state grid in
`tests/conftest.py`, with train n=120 and eval n=120 on disjoint, enforced seed
sets (overlap raises):

| intent | uninformed AUC | informed AUC |
| --- | --- | --- |
| `sync_fork_with_upstream` | 0.500 | 0.962 |
| `restore_submodule_state` | 0.500 | 0.945 |
| `keep_uncommitted_work_and_sync` | 0.500 | 1.000 |
| `follow_renamed_upstream_branch` | 0.500 | 1.000 |
| `sync_through_a_conflicting_lockfile` | 0.500 | 1.000 |

**On uninformed requests the text cannot carry the resolution (AUC 0.500), by
construction and not by measurement: the sampler never consults state, so every
state receives the same text for a given seed, every positive has a negative with
an identical score, and the AUC is 0.500 for any classifier and any phrasing
list — including a deliberately leaky one. The test on that regime is a plumbing
tripwire: it fails if the sampler starts consulting state, the regression that
would restore the original flaw, and it cannot certify the phrasing
distribution. On informed requests the wording nearly determines the resolution
(AUC ≈ 0.95, measured by `bench/textcontrol.py`), which is the boundary condition
where the mechanism is not needed.** An earlier draft pooled the two regimes into
one number (0.795–0.801, the pooled figures from the control's pre-split
revision) that described neither — it averaged a regime in which the text is the
answer in disguise with one in which it is noise — and that is why the split
exists.

The uninformed regime is the one gated by `LEAKAGE_CEILING`, as a tripwire for
state reaching the sampler rather than as a leak detector; the informed AUC is
reported as the boundary condition and never gated. The per-intent AUCs must be
reported together with the informed/uninformed mixture of the pairs, because that
mixture is what makes the numbers interpretable.

### Out of scope for the family

Anything requiring credentials, network access to real GitHub, or judgement
("tidy up this README"). Every fault is injected locally, solved locally, and
verified locally.

---

## 4. Architecture

Boundaries matter more than internals here: the experiment is only interpretable
if arms differ in *exactly one* component.

```
                    ┌──────────────────────────┐
                    │  bench/run.py            │  episodes × arms
                    └────────────┬─────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
      ┌───────▼──────┐   ┌───────▼───────┐  ┌───────▼────────┐
      │ agents/react │   │ agents/       │  │ agents/        │
      │  (arm 1)     │   │ dispatch      │  │ dispatch       │
      │              │   │ semantic      │  │ preconditions  │
      │              │   │  (arm 2)      │  │  (arm 3)       │
      └───────┬──────┘   └───────┬───────┘  └───────┬────────┘
              │                  │                  │
              │            ┌─────▼──────┐           │
              │            │ library.py │◄──────────┘
              │            └─────┬──────┘
              │                  │ programs
      ┌───────▼──────────────────▼───────────────────────┐
      │ runtime/replay.py + runtime/guard.py             │
      │   ▲                                              │
      │   │  MUST NOT be able to reach provider.py        │
      └───┼──────────────────────────────────────────────┘
          │
   ┌──────▼───────┐   ┌──────────────┐   ┌────────────────┐
   │ sandbox.py   │   │ tasks/faults │   │ provider.py    │
   │ throwaway    │   │ seeded       │   │ the ONLY place │
   │ repos        │   │ injectors    │   │ that calls LLM │
   └──────────────┘   └──────────────┘   └────────────────┘
```

**The only difference between arms 2 and 3 is which dispatch function calls into
the library.** Everything downstream — admission, replay runtime, guard, ground
truth checker, ledger — is shared code. A difference in outcome is therefore
attributable to dispatch and not to the rest of the pipeline. This is a design
constraint, not a description of the current layout: if arms 2 and 3 ever need
different runtime code, the comparison has been compromised and the design has
failed.

| module | responsibility | depends on |
| --- | --- | --- |
| `program.py` | the `Program` model: intent, parameters, preconditions, body, postconditions, provenance, status | nothing (pure data) |
| `provider.py` | the only LLM boundary; returns real token usage | httpx |
| `signatures.py` | how a task and its environment state are described | — |
| `sandbox.py` | build/destroy disposable repo + bare upstream | git |
| `tasks/faults/*` | seeded fault injection, task text, ground-truth checker | sandbox |
| `library.py` | storage, two-sided admission, both dispatch strategies | program |
| `agents/react.py` | arm 1: the ReAct-style baseline (Yao et al., ICLR 2023) — observe → act → observe | provider, sandbox |
| `agents/compile.py` | solved task → candidate program; admission gate | provider |
| `agents/dispatch.py` | arms 2 and 3 — **the experiment** | library |
| `runtime/guard.py` | screens model-authored bodies before execution | — |
| `runtime/replay.py` | executes programs, re-checks postconditions; **cannot reach `provider`** | guard |
| `bench/ledger.py` | the episode record | — |
| `bench/run.py` | episode scheduling, repeat structure | all of the above |
| `bench/report.py` | ablation table, cost curve, mismatch comparison | ledger |

### Data flow of one episode

```
seed ──▶ fault.inject ──▶ sandbox ──▶ fault.task_text ──▶ request
                                            │
                    ┌───────────────────────┴────────────────────┐
                    │                                            │
              arm 1: ReAct loop                    arms 2/3: dispatch
              (LLM every step)                            │
                    │                        ┌────────────┴────────────┐
                    │                    hit │                         │ miss
                    │                        ▼                         ▼
                    │                 replay(program)          solve with LLM
                    │                 (ZERO tokens)                    │
                    │                        │                  compile → admit
                    │                        │                         │
                    └────────────────────────┴─────────────┬───────────┘
                                                           ▼
                                             fault.check ──▶ outcome
                                                           ▼
                                                    ledger.append
```

---

## 5. The `Program` artifact

```yaml
id: sync-fork-dirty-tree
intent: bring a fork into sync with upstream while preserving uncommitted work
parameters: [work_dir, upstream_remote, upstream_branch]

preconditions:                      # executable probes; ALL must hold to fire
  - name: has_uncommitted_changes
    probe: 'test -n "$(git status --porcelain)"'
  - name: upstream_is_ahead
    probe: 'test "$(git rev-list --count HEAD..{upstream_remote}/{upstream_branch})" -gt 0'
  - name: not_mid_rebase
    probe: 'test ! -d .git/rebase-merge'

body: |                             # model-authored; runs unattended later
  git stash push -u -m pl-sync
  git fetch {upstream_remote}
  git rebase {upstream_remote}/{upstream_branch}
  git stash pop

postconditions:
  - name: upstream_contained
    probe: 'git merge-base --is-ancestor {upstream_remote}/{upstream_branch} HEAD'
  - name: no_residual_changes
    probe: 'test -z "$(git status --porcelain)"'

provenance:
  compiled_from_task: "My local uncommitted work must survive, and this fork needs to be in sync."
  model: deepseek-flash
  episode_id: react/dirty_tree/seed-12
  fault: dirty_tree          # the family whose states this program serves
status: candidate            # not yet admitted; see the caveat below
```

Preconditions are **shell probes**, not prose and not Python callables. Probes
survive serialization into the committed library, are reviewable in a diff, and
can be executed by the runtime without importing program code. A reviewer can
verify a dispatch decision by hand, which would not be true of a similarity
score.

Parameters keep one program applicable to many repositories: the library stores
*knowledge*, not a transcript of one repository's history.

**Illustrative caveat:** the YAML above is a shape, not an admitted program. The
body in particular is not claimed correct — `git stash pop` after a rebase can
conflict, and a real program would have to handle that. No program has yet been
through admission.

### Lifecycle

```text
candidate ──admission passes──▶ admitted ──postconditions fail──▶ demoted
(stored, status=candidate)         │
    │                              │ two wrong-variant fires
    └──admission fails──▶ stays    ▼
         stored (candidate;    quarantined
          rejection recorded)  (withdrawn from dispatch; retained for analysis)
```

The two mismatches are counted over the ledger's own definition: a fire whose
`fired_variant` is not the state's ground truth, *whether or not the episode
succeeded*. That is why the count is needed at all — a program that fires the
wrong resolution and still satisfies its postconditions is never demoted, and a
demoted program is already withdrawn (both matchers return only `admitted`), so
it can never accumulate the second mismatch. The load-time variant check is the
other route into `quarantined`. It keys on `provenance.fault`, not on `intent`: a
compiled program's intent is prose, so a check keyed on it would protect only the
hand-written artifacts. `Library.load_all` applies the verdict in memory and
writes nothing — a read that reparsed the library must not mutate it, and
`library_hash` loads every program — while `Library.quarantine_undeclared()` is
the explicit write that records the status and its reason (issues #66, #69).

Nothing is deleted. A program that mis-fired is the evidence for Claim 2's
numerator, and removing it would erase the result. A candidate is written to the
library **before** the gate runs, so a program that fails admission keeps its
`status=candidate` and stays as a rejected record rather than being removed. That
rejection is itself evidence — the mismatch analysis needs it — and the episode
keeps its token cost.

---

## 6. Admission: the two-sided gate

A program becomes replayable only when **both** halves of its contract are
demonstrated:

```
DECLARED   before any sandbox: every `{name}` the body uses is named by at least
           one precondition's probe. A precondition is not only how a program
           decides *whether* to fire; it is how the program declares what it
           needs, and a state it may fire in need not bind an undeclared
           parameter. A body parameter no precondition names -> not admitted,
           with the parameter named (issue #77).

POSITIVE   on N freshly faulted sandboxes: body runs, postconditions hold.
           Failure -> not admitted.

NEGATIVE   on sandboxes it must NOT claim, the preconditions must REJECT it.
           Three classes are built, from cheapest and broadest to narrowest:

             fault-free          a clean worktree with upstream already merged
                                 (`build_sandbox(seed, [])`): nothing needs doing,
                                 so every program must refuse it. A precondition
                                 set that accepts it is a DEFECT, not a
                                 convenience.
             unrelated fault     every other fault's injected states: one seed
                                 per distinct state the injector can select
                                 (`FaultSpec.variant_for_seed`), so a fault with
                                 several states is probed in each rather than at
                                 one fixed seed -- states this program's intent
                                 has nothing to do with. Except an OVERLAP state:
                                 one this intent's own decision rule labels (e.g.
                                 `lockfile_conflict`'s diverged branch, which the
                                 sync intent calls `merge`). It is judged like a
                                 sibling: firing is correct if the program's
                                 resolution is one the state accepts, a mismatch
                                 otherwise (issue #158, ADR-0019, ADR-0023).
             sibling resolution  the program's own fault, injected at a seed whose
                                 state a *different* resolution of the same intent
                                 is the label. Firing there is a mismatch unless the
                                 state also accepts the program's resolution -- the
                                 fault's checker passes it there -- and then it is a
                                 correct fire that counts toward the breadth cap
                                 (ADR-0023: merge and rebase on any diverged state).

           Each rejection names the class that rejected it and the number of
           states that class checked, so a reader can tell how much of the gate
           actually ran. A program that accepts any of these fires where firing is
           wrong, which is exactly the mismatch failure this project measures.

BREADTH    a measured cap over the whole sampled universe (clean plus one sandbox
           per distinct state of every fault): the fraction of states the
           preconditions fire on must not exceed `BREADTH_CAP` = 0.50 (issue #10).
           A set that matches most states is not targeted. Measured, every gold
           program fires on one of the ten sampled states (0.10). The three
           enumerated classes above are a sharpening of this: after they reject
           the clean, unrelated and sibling states, only the program's own fault's
           remaining states can still fire, so for the current family the cap does
           not bind -- it is the holistic backstop, and the recorded breadth
           number (`agents.compile.precondition_breadth`), that become load-bearing
           if a fault injects a state no enumerated class names.
```

The negative half is what makes Claim 2 testable. Without it, arm 3's advantage
would be an artifact of admitting programs freely and merely hoping their
preconditions were specific. Admission failures are recorded with reasons: a
high rejection rate is a finding about the compile step's ability to write
specific preconditions, not an inconvenience to be suppressed.

---

## 7. Measurement

### Ledger

One JSONL line per episode; every number reported is a grouping over this file.

```
{arm, task_id, fault_type, occurrence_index, occurrence_role, seed,
 tokens_in, tokens_out, uncached_tokens_in, cached_tokens_in, cache_write_tokens_in,
 llm_calls, tool_calls, wall_clock_s,
 outcome: success|fail|fallback|refusal|invalid, timed_out,
 correct_variant, fired_variant, ground_truth_ok, recorded_state_intact,
 program_id, dispatch_score, admitted,
 refusal_reason, compile_failure_reason, invalid_reason, replay_failure_reason, model,
 temperature, admission_gate, rerank_k}
```

- `occurrence_index` — 1 for the first time this fault type is seen, 2 for the
  second, and so on. Grouping by it produces the headline curve.
- `occurrence_role` — `variant` for the first occurrence of an **instance** of
  this fault, `replay` only on a genuine repeat of one (ADR-0005; before it the
  key was the resolution, which made same-resolution seeds — byte-identical
  environments — one observation counted many times). Declared by
  `bench/splits.py` from the injector's own seed-to-instance mapping and written
  onto every row by
  `bench/run.py`; required, with no default. **The two analyses need different
  occurrences.** The cost curve is only meaningful on the replays, because a
  variant is the first sight of an instance and no program can have been admitted
  from it yet — the curve bends exactly where an instance recurs. The mismatch
  comparison is only meaningful on the variants, because a replay's instance was
  introduced by an earlier occurrence and the program that answers it was
  admitted there, so counting a replay counts one observation twice. It records
  the plan's role and not what happened: a row labelled `replay` may still have
  paid full price, because no program was admitted for its state or the one that
  fired was refused. `llm_calls`, `outcome` and `fired_variant` say what
  happened. A reader who does not know this will compute the curve over the
  occurrences where it cannot bend, or an interval over one observation counted
  many times.
- **Input tokens are metered in three components, and none of them is a total.**
  `uncached_tokens_in` is what the provider billed at the full input rate,
  `cached_tokens_in` is the **cache-read** component, and `cache_write_tokens_in` is
  what was written to the cache. `tokens_in` is the provider's own reported total —
  their sum — and is kept so the split can be checked against it, **never as a billing
  basis**: DeepSeek publishes cache-hit input at 1/50th of cache-miss input, so pricing
  the total at one rate overstates any arm that caches, and arm 1 is the arm whose
  prompt grows with the transcript. Derived as: `prompt_cache_miss_tokens` when the
  provider publishes it (DeepSeek does), otherwise `prompt_tokens - cache hits`,
  saturating so a provider reporting more hits than prompt tokens cannot make it
  negative. **Per provider:** OpenAI-style APIs report `prompt_tokens` *including*
  cached tokens, so it is never the uncached count; Anthropic-style APIs report the
  cached portions as their own fields and bill a separate cache write, which is why
  `cache_write_tokens_in` exists even though DeepSeek always writes zero. Pricing the
  three at published rates is a separate step and is not implemented — it needs a rate
  table, and DeepSeek's rates also differ by peak and off-peak hours, so a correct
  figure needs the rate (or the call time) recorded per episode rather than applied
  afterwards.
- **Correctness is derived, not stored.** `correct_variant` is the ground truth for
  the state (from the intent's decision rules, in separate code from the programs'
  probe strings), `fired_variant` is what the program that ran actually resolves,
  and `ground_truth_ok` is whether the environment reached the expected state.
  `misfired` and `succeeded` are computed from those, so the quadrant that matters —
  a wrong fire in an episode that nevertheless succeeded via the fallback — is
  expressible. A stored outcome could not hold both halves without the two
  disagreeing, and `outcome` therefore records only how the mechanism completed.
- **Spec-gaming is derived from a fact, like the other correctness verdicts.**
  **A checker's selector comes from the harness, not from the clone.** A fault with
  more than one injected state needs to know which clause applies, and `submodule_moved`
  used to read that from a recorded `refs/sandbox/submodule-state` — a plaintext label
  inside the environment the graded code reads, so a precondition could be
  `test "$(git cat-file -p refs/sandbox/submodule-state)" = "remove"` and dispatch
  correctly without diagnosing anything (issue #103). The state now travels on the
  `Sandbox` object, which the harness owns, and the clause is still read from the
  environment: the selector says *which* clause, the environment decides *whether* it
  holds. The same rule covers the values and the harness's own bookkeeping: a fault's
  pre-injection tip, its recordings, the double-injection marker, the submodule path the
  probes bind `{submodule_path}` from, and the post-injection ref snapshot all live in
  `Sandbox.recorded`, which the harness owns. The clone holds nothing under
  `refs/sandbox/` (issue #161, ADR-0021): a ref there holds in every sandbox and in no
  real repository, so a precondition could anchor on it and pass admission.
- `recorded_state_intact` records whether the resolution left the **recorded state**
  alone — today that the work repository was not re-cloned or wiped (every object its refs
  named at the start is still in the store) and upstream's history (`tasks/invariants.py`, checked after the
  fault's own clause passes and recorded on every row it ran for). A row that
  reached the expected state while `recorded_state_intact` is `false` was **spec-gamed**:
  it repaired the fault destructively, satisfying the graded predicate by an
  unintended route. That is its own column in the ablation table and its own
  pooled line in the report, because such a row otherwise carries
  `ground_truth_ok=false` and reads exactly like a resolution that simply failed
  to repair the fault. `null` means the check did not run — an invalid episode, or
  a row written before it existed — and is never counted as gaming. The field was named
  `refs_intact` until issue #96 renamed it; a revision-history row below still uses the old
  name because a record of a time is not rewritten (rule 5).
- **Denominator rule.** `fail`, `fallback` and `refusal` all carry
  their token spend into the means — an episode that crashed after 4,000 tokens
  still cost 4,000 tokens. `invalid` is the sole exception: the episode never
  ran, so it is counted and reported as its own rate but excluded from metric
  denominators. See item 7 below.
- `cached_tokens_in` is separate from `tokens_in` because provider-side prompt
  caching can make the ReAct baseline look cheaper than the work it performed.
  Arm 1's prompts grow with the transcript, so caching flatters it most; the
  confound is recorded so it can be inspected rather than assumed away.
- `model` is recorded per episode because provider-side model drift between
  runs is a confound that silently invalidates a comparison.
- `admission_gate` is the gate the dispatched library's programs passed —
  `two_sided` or `positive_only` — the 2×2's admission factor (ADR-0010).
- `rerank_k` is how many first-stage candidates arm 2's reranker rescored, `null`
  when arm 2 ran single-stage (ADR-0011). With a reranker the threshold judges the
  reranker's score, so a row's `dispatch_score` and `similarity_threshold` are read
  with it.
- `temperature` is the sampling temperature the episode's model calls were sent
  with, read off the completions rather than from run configuration (issue #6).
  The provider sends 0 unless overridden. `null` means no call reported one — a
  replay, which consults no model. It records what was **sent**: whether the
  endpoint honours it has not been checked against a live call, and 0 is not a
  determinism guarantee on a hosted model.
- `dispatch_score` (arm 2) and `admitted` are recorded so a tuned comparison is
  distinguishable from an untuned one.
- The four reason fields are separate because they are four different signals.
  `refusal_reason` holds the guard's reason and is set only when `outcome` is
  `refusal`, so the guard refusal rate is a grouping over that one field and
  cannot pick up anything else. `compile_failure_reason` holds why a compile or
  admission produced no usable program (a malformed reply, a failed gate, a
  program id collision). `invalid_reason` holds why an episode could not be graded at
  all. `replay_failure_reason` holds why a program that *fired* could not be run
  at all — today, a body naming a declared parameter the environment cannot bind,
  so no command executed and there is no postcondition evidence (issue #76); a
  body that ran and failed its postconditions is recorded by the demotion
  instead. A single field carrying all four would silently fold compile failures
  into the safety metric — the defect issue #60 fixed.

### Design

The primary measurement is a **dispatch-level benchmark**, not an episode run:

```
  labelled (repo-state, candidate-program) pairs, split by fault seed
        ├─ admit set    build the precondition vocabulary
        ├─ tune set     calibrate arm 2's representation and threshold
        └─ eval set     the reported numbers
        │
  arm 2 sweeps its threshold; arm 3 is one operating point (ADR-0008)
        │
  mismatch-vs-coverage curve, Wilson intervals, power statement
```

Each pair is labelled by mechanically deriving the expected state from the
fault-injection spec (§3), with ground truth kept in separate code from the
admission postconditions (§6) and validated by two negative controls: a no-op
agent must fail every fault, and a deliberately wrong program must fail too.

A dispatch decision costs no LLM call — arm 3 evaluates predicates, arm 2 scores
text similarity, lexical today and an embedding model behind the same seam
tomorrow — so the pair count is chosen to make the comparison adequately powered
rather than to fit a budget.

**The episode loop is retained only as a demonstration, explicitly underpowered:**

```text
5 measurable faults × 4 occurrences × 3 arms = 60 episodes   (demo, UNDERPOWERED)
    (the nominal grid is now runnable: since ADR-0029 every fault has a
     registered intent and EXCLUDED_FROM_BENCHMARK is empty)
```

It is labelled underpowered wherever it appears and excluded from the primary
claim. Its value is showing the harness works end to end, not producing a result.
The 60-episode figure is the nominal grid, and since ADR-0029 the runnable demo is
all of it (24 before ADR-0027 registered `dirty_tree`, 36 before ADR-0028, 48 before
ADR-0029). An earlier draft stated the nominal figure as though it could run before it
could.

Seeds are split into disjoint admit / tune / eval sets and every arm is routed
through the same (fault, seed) pairs, so the comparison is paired rather than
between-populations. The unit of analysis is the **environment** — a resolution
together with the drawn Tier 1 axis values, the instance identity
`FaultSpec.instance_for_seed` names — not the seed and not the episode. An
occurrence is a **variant** the first time its *instance* is seen and a **replay**
only on a genuine repeat of one (ADR-0005; before it the key was the resolution,
because same-resolution seeds built byte-identical environments). The two roles
feed different analyses — the mismatch comparison over the variants, the cost
curve over the replays (see "Figures"). Since ADR-0005 the injectors draw each
state shape's file bodies, local commit count and subjects, and conflict hunk
positions per seed, and `build_sandbox` refuses a draw whose observed resolution
is not the one declared — so a new seed is usually a new environment, and a
mislabelled one cannot enter the comparison. The axes beyond Tier 1 (file names,
submodule paths, branch names) remain issue #6's, because they need probe and body
parameter binding first.

### Figures

**Figure 1 — mismatch vs coverage.** The primary figure. Arm 2's curve over every
distinct acceptance threshold, and arm 3 as a single operating point, since it has
no score to sweep (ADR-0008); mismatch rate against coverage, Wilson intervals at
each operating point, and the pre-registered coverage point — arm 3's own — marked.
Produced by `bench.primary.write_primary_report` (`figure1.csv`, the deliverable, with
every point's numerator, denominator and interval; `primary.txt`; `figure1.png`), one
report per request regime. The report states the matched comparison and the item-3
power statement, unless `vacuous_reason` fires, in which case it states that instead
and draws the curve without a comparison. The #7 dispatching baselines may join the
plane as context (2b as a curve, 2c as a point). The outcomes come from
`bench.library_pairs.library_pair_outcomes`, which builds each pair's sandbox and
dispatches it through the compiled library's own matchers, so the pair-level arms are the
arms the episodes run; ADR-0022 records which pairs it builds.

**Figure 2 — cost vs repeat (secondary).** **Cumulative amortized** tokens and LLM
calls per episode against `occurrence_index`, one line per arm, from the
underpowered demo and **over its replay occurrences only**, with the break-even
marked on the figure and stated in the report text. The accumulation is the
figure's subject, not the per-occurrence mean: compile cost is charged to
occurrence 1, so a compiled arm starts above the baseline and can only cross it
later — or not at all — and a mean per occurrence is a snapshot that cannot show
which. Where no crossing is computable, because the arms share no occurrence index
or because one never crosses, the report says which of those it is rather than
leaving a blank. A variant occurrence is the learning pass — no program admitted
from a state the run had not yet seen can exist — so a curve that included the
variants would average the cost of learning a state into the cost of replaying it.
Arm 1 pays full price on the same occurrence indices, because it has no library,
and that contrast is the comparison the figure exists for. Reported as a cost
model, not as a contribution; the figure and its CSV name the occurrences used.

**Figure 3 — the admission factor.** The 2×2 {dispatch: semantic|precondition} ×
{admission: gated|ungated} comparison, which separates the negative-sandbox
criterion (Claim 3) from predicate dispatch (Claim 2). *Ungated* is the
positive-only gate — every admission check except the negative sandboxes — and
both libraries come from **one compile gated twice** (`build_library(positive_only_root=...)`,
ADR-0010), so they hold the same programs. Each row records its `admission_gate`, and
`bench.report.admission_factorial` gives one cell per (arm, gate); the
{precondition, positive-only} cell is issue #4's arm 3-prime.

### Pre-registered analysis

**Revised 2026-09-13, before any data exists.** The original pre-registration
named an episode-level paired mismatch difference as primary. A method review
found that metric degenerate as designed (ADR-0001). The revision is recorded
rather than silently applied: no episode has been run, so no analysis was chosen
after seeing results.

1. **Primary metric:** the mismatch-versus-coverage curve, and the difference
   between dispatchers **at matched coverage**. The directional claim is stated at
   a pre-registered coverage point; the full curve is reported regardless.
   Defined by ADR-0008 and computed by `bench.coverage`: *coverage* is fires over
   pairs, negatives included; *mismatch* is wrong fires over fires, a fire on a
   negative pair counting as wrong. A fire is wrong when its resolution is not one
   the state **accepts** -- its label, or another resolution its fault's checker
   also passes there (ADR-0023) -- so `rebase` on a `merge`-labelled diverged state
   is not a mismatch, and `discard` on any diverged state with real local work is. Arm 2's threshold is swept over every distinct
   score; arm 3 has no score and is one operating point, and **its coverage is the
   pre-registered coverage point**. Arm 2's point nearest it is the matched one, a
   tie going to the lower coverage, and the coverage gap is reported beside the
   difference. The regimes are compared separately, never pooled (ADR-0004). While
   arm 3 decides every pair correctly — hand-written programs that agree with the
   labelling rules do, by construction — the comparison is vacuous, and the report
   prints `vacuous_reason` instead of it.
2. **Unit of analysis:** the environment -- the instance a seed builds (ADR-0005) --
   not the episode. Seeds split into disjoint admit / tune / eval sets; episodes
   paired by (fault, seed). Within the episode loop the independent observations are
   its **variant** occurrences only: a replay re-asks a state an earlier occurrence
   introduced, and is excluded from every independence claim (see "Ledger"). The plan
   is run as **k = 3 whole-run replicates** (`bench.run.run_replicates`; `bench.live
   --replicates 3`, revision 77), and
   seed-level uncertainty comes from `bench.report.cluster_bootstrap`, which resamples
   instances with all their arms and replicates, so a repeat never counts as a new
   observation (issue #6, ADR-0012).
3. **Power statement:** required — the detectable effect size at the chosen pair
   count, coverage points, and alpha. `bench.coverage.detectable_difference` computes
   it for two arms' fire counts (two-proportion normal approximation, α = 0.05
   two-sided and power 0.80 by default; ADR-0008 records those as defaults, not as a
   registered α for the directional claim).
4. **Secondary metrics:** tokens and LLM calls per episode by occurrence index,
   accumulated into a running amortized cost with the break-even marked (the cost
   model, Figure 2); **arm 2's embedding currency separately**, never added to the LLM token fields;
   **the arm triple** — success rate, tokens per episode and tokens
   per success, each with its denominator — reported with a **Pareto frontier over the
   three** (`arm_triples.csv`, `pareto.png`), so that cheaper-per-episode, cheaper-per-
   success and more-successful are read together rather than one at a time, per item 8;
   **the issue #7 baselines on that same frontier**, the first of which is the zero-token
   **gold oracle floor** (ADR-0014): it replays the ground-truth variant's hand-written
   resolution, so its success bounds what any arm could reach and its cost bounds the
   cheapest an arm could be, and because it consults ground truth it is an oracle rather
   than a dispatcher and is excluded from the matched-coverage comparison; and **arm
   2c, intent-key dispatch** (ADR-0015), a lookup on the normalised request string that
   abstains when its key's programs disagree -- a dispatcher with no score, so one
   operating point, measurable on the pairs by `bench.coverage.intent_key_outcomes`;
   **arm 2b, a soft vote over arm 3's own probe features** (ADR-0016), whose learned
   threshold is recorded on its rows and whose tie with arm 3 decides item 12; and
   **arm 1b, ReAct with a memory of its own past successes** (ADR-0017), the
   experience-reuse baseline for the cost curve, which runs in the online mode only and
   records how many stored trajectories each episode's prompt carried (`memory_recalled`);
   **the raw character length of each prompt prefix the run sends**, reported once
   rather than per arm because every arm sends the same two prompts — arm 1b's recalled
   memory goes in the user message, not the system prefix — and the arm-level difference
   is how much text accumulates around the prefix, which the input tokens carry; compile success rate; fallback rate; guard refusal rate. The
   episode loop is labelled underpowered and excluded from the primary claim.
5. **Tuning discipline:** arm 2's representation and similarity threshold are
   tuned on the tune set, disjoint from eval, and the value used is written into
   every ledger line. An untuned control arm would be a straw man; a control arm
   tuned on its own evaluation set would be cheating. The record shows which was
   done.
6. **Extension rule:** if the interval on the primary difference spans zero at the
   pre-registered coverage point, extend the pair count rather than re-analysing
   the existing set differently. If it still spans zero, the result is "no
   difference detected at this N", with the interval shown.
7. **Failed episodes stay in the denominator**, with their partial token spend.
   Dropping them would make amortization look better than it is.
8. **Cost per success is reported alongside cost per episode**, because an arm
   that succeeds more often may legitimately spend more per episode.
9. **Invalid episodes are excluded from denominators but never hidden.** An
   episode that could not run (sandbox or infrastructure failure) is recorded as
   `invalid` and reported as a rate. **An invalid rate above 10% makes the run
   suspect**, and it is re-run rather than analysed — infrastructure flakiness
   that removes episodes non-randomly is indistinguishable from a real effect.
10. **"Equal success rate" requires an equivalence test; otherwise the text says
   "comparable".** The words are not interchangeable: failing to detect a difference
   is not evidence of none, and a run this size will usually fail to detect one. The
   report runs a **TOST** on arms 2 and 3's pooled success rates at a
   **pre-registered margin of ±10 proportion points and α = 0.05** — registered here,
   before any data, because a margin chosen after seeing the interval can be widened
   until equivalence passes. It reports the difference, the 90% interval, both
   one-sided p-values, and which of three cases holds: equivalent within the margin;
   outside the margin, so the arms differ by more than it; or too wide to decide,
   which is an underpowered run and **not** a finding that the arms differ. The
   verdict is produced by `bench/report.py` (`success_rate_wording`) rather than left
   to whoever writes the prose, and its variance uses Agresti–Coull-adjusted
   proportions, because the raw Wald variance is zero at a 0%/100% rate and would let
   a single episode's zero-width interval pass as equivalence.
11. **Arm 2 must clear a floor before it counts as the baseline.** Before any eval
   episode, arm 2's strict top-1 is measured on the **informed** regime over the
   **tune** seeds, against the candidate texts of the library the eval dispatches with
   (`bench.similarity_probe.tune_baseline`). Arm 2 is a usable baseline only if the
   **95% Wilson lower bound** of that rate exceeds chance, 1/3 for three candidate
   resolutions (`bench.report.arm2_baseline_floor`). The bound is judged, not the point
   estimate, because one pair above chance is noise; at 24 decidable pairs the floor
   falls between 12 and 13. Below it, a report may not say that precondition dispatch
   beats text similarity. It still reports the comparison, with the floor's verdict
   beside it, because a failed baseline is a result about arm 2 and not a reason to
   drop the row. Registered here, before any data, for the same reason as item 10's
   margin (issue #116, ADR-0007). `bench.live` measures it on the two-sided library
   before any eval episode and reports it beside Figure 1 (`baseline_floor`,
   revision 76). Figures on the probe's seeds 0–3 and on hand-written gold are
   exploratory and are never judged by this floor.
12. **Claim 2 is restated if a soft vote over the same probes ties arm 3.** Arm 2b
   (issue #7, ADR-0016) scores each admitted program by the fraction of its
   preconditions that hold, the same per-predicate results arm 3 reads, with a
   threshold learned on the **tune** seeds for most correct decisions (ties to the
   stricter floor; `bench.soft_vote.learn_soft_threshold`). On the eval pairs its
   swept curve is matched to arm 3's coverage by item 1's rule, and the two per-fire
   mismatch rates are compared by item 10's TOST at the same ±10pp margin and α.
   **Equivalent** means requiring every probe adds nothing measurable over the same
   information, and Claim 2 is then stated as *"probe-based dispatch beats
   text-similarity dispatch"*. A matched 2b point at threshold 1.0 *is* arm 3 and
   makes its decisions exactly, so that tie is reported as **collapsed** and keeps the
   registered wording (`bench.soft_vote.claim2_verdict`). Registered before any eval
   data.
14. **Arm 2's fourth scorer is a judge that reads the state, and it is gated like any
    other baseline** (issue #264, ADR-0034). `bench.judge_similarity.JudgeSimilarity`
    sits behind the `Similarity` / `ScoresMany` / `ReportsUsage` seam, so nothing
    downstream of the seam learns what it is and **arm 3 is unchanged by construction**
    (it does not read the seam). Four things are registered with it. *What it sees:*
    exactly what arm 2 sees -- the request, `StateFingerprint.as_text()`, each
    candidate's `_program_text` -- and **not** the probe strings, because a judge that
    reads the probes reasons about the predicates arm 3 executes, which is a different
    mechanism and a separate experiment. *How it is called:* one provider call per
    `(request, state)` pair through `score_many`, returning one score per candidate, so
    item 1's sweep still applies; `__call__` stays available and memoised for the
    probe's pairwise loop. *What makes it reproducible:* its per-pair scores are
    committed as a JSONL cache under `results/`, `bench.rescore --judge-cache` recomputes
    a figure from that file with **no key and no tokens**, and a cache whose prompt hash,
    model or per-record candidate count disagrees with the running configuration is
    **refused** rather than reused. *What stays in the denominator:* an empty completion,
    a transport error, an unparseable reply, a wrong length and an out-of-range score all
    become an **abstain** (`0.0` for every candidate) and are counted separately;
    `ProviderAuthError` is re-raised and stops the run (issue #157) and there are no
    silent retries. **The floor comes first** (item 11): the judge's informed-regime
    strict top-1 on the tune seeds against each frozen library decides whether it is a
    usable baseline, and below it no "beats text similarity" sentence may be written.
    **Repeatability is registered, not assumed:** a pre-registered subsample is asked
    `k >= 2` times and the agreement is reported beside the judge's row, with the
    statement that the three text scorers carry no such component -- a judge is an LLM,
    and the primary metric moved to pair level precisely because sampling variance made
    the episode-level comparison unreadable (revision 41 records that temperature 0 is
    not determinism on a hosted model). Its spend is `SimilarityUsage`'s own currency,
    recorded in the rescore's record beside the scorer's identity, never folded into the
    LLM's `tokens_in`/`tokens_out` nor into an episode's `embedding_*`; the budget is
    declared here before the run and an exceeded budget stops it. **Registered before the
    first judge call:** no judge has been called, the floor and the three rescores need a
    provider key the caller supplies, and nothing here reports a number.

15. **A hybrid dispatcher is measured as a family, and read at arm 3's coverage** (issue
    #265, ADR-0035; registered as item 15 because items 13 and 14 are the held-out stage and the
    judge, whose changes are open PRs this branch predates). `bench.hybrid` builds the family from
    the pair pass's own record -- each environment's accepted program ids and each request's
    complete text ranking, which `bench.library_pairs` now keeps instead of discarding -- so no
    sandbox and no model call is added. Its definition is fixed: rank the admitted programs by the
    text scorer, unthresholded; keep those at or above a retrieval floor and within the top `k`;
    fire the **best-ranked one whose preconditions all hold**; otherwise fire nothing. The grid is
    registered as `k` in `(1, 2, 3, 5, 10, None)` (`None` = the whole ranking) crossed with every
    distinct retrieval score plus no floor, and changing either is a revision logged here first.
    **The claimable comparison is the point nearest arm 3's coverage** (a tie to the lower
    coverage, item 1's rule), reported with its Wilson interval, the coverage gap and the smallest
    detectable difference at those fire counts; every other point is the family, not the hybrid's
    number. **Two readings are registered with it.** *Selection is the trap:* two knobs over a few
    hundred points on 400 pairs makes the best cell a maximum over many draws, so the widest
    zero-mismatch point per `k` is a **description of these pairs**, bounded by the 95% Wilson upper
    limit of the mis-fires it did not have, and never an unbiased estimate of the rate at a floor
    chosen because it looked good. *The gate's refusals are a property, not a finding:* the hybrid
    cannot fire on a negative pair because the gate is applied per candidate and a fault-free
    sandbox accepts nothing, so that is its definition rather than a result. A named operating
    point's `k` and floor are chosen on the tune seeds and the record says where; a sweep is not
    tuning and needs none. The stage reads a committed frozen library read-only, with no model call
    and no key.

13. **The primary metric is also measured on held-out states, in its own report.** The
    states item 1's pairs are built from come from the same injectors admission tests
    against, so the figure is an in-distribution one; `bench.heldout` measures a
    **held-out block** -- `bench.splits.HELD_OUT_SEEDS`, 40 seeds at 3000-3039, where
    `dirty_tree`'s injector draws a path set no shipped seed draws
    (`tasks.spec.HELD_OUT_SEED_BASE`) -- against a **committed** frozen library, read-only
    (its digest is compared before and after), with no model call and no rebuild. The
    metric, the regime and the coverage point are item 1's; the report is written to a
    fresh directory and is **never pooled with the eval pairs**, because the two answer
    different questions. **The unit stays the environment** (item 2): the block builds 31
    distinct instances across its 40 seeds, `bench.heldout` reports that count beside the
    seed and pair counts, and a rate or interval is read at it rather than at the pair
    count -- arm 3 is request-blind, so the requests crossed with one environment carry no
    second observation. Two readings are registered with it. **No win wording:** item 11's
    floor is measured on the tune seeds against the shipped states, so it does not transfer
    to states no library saw, and the stage carries `bench.heldout.WORDING_LIMIT` for that
    reason; below the smallest detectable difference item 3 computes for the achieved fire
    counts, nothing may be read. **A vacuous comparison is an outcome, not a failure:**
    where arm 3 decides every held-out pair correctly, `bench.primary` prints
    `vacuous_reason` and no comparison; on hand-written gold that is a tautology, but on
    **compiled** programs it means arm 3 neither mis-fired nor abstained, so the interval on
    its held-out rate is the readable statement and the arm-2 contrast waits for a block on
    which arm 3 abstains somewhere. Registered before any held-out state was built
    (issue #263, ADR-0033); the paths, the pool, the block and the wording limit are
    pre-registration values, so changing one is a revision logged here first.

#### The seed plan and the run invocation

The split is fixed in `bench/splits.py` before any data exists, because a split
chosen after seeing scores is not a split, and the ledger records `seed` on every
row, so a reader must be able to tell which set an episode came from.

```text
smoke  0,1,2,4     4 seeds   shake out the pipeline; admit the programs (the admit set)
tune   1000-1015   16 seeds  calibrate arm 2's similarity threshold (item 5)
eval   2000-2039   40 seeds  the reported numbers
held-out 3000-3039 40 seeds  states no library saw; measured apart from the eval pairs
                             (item 13, issue #263)
```

**The admit set builds one frozen library** (issue #4, ADR-0009). `bench.build_library`
solves each admit-set `(fault, seed)` against its own empty scratch library -- so nothing
is dispatched while the artifact is built -- compiles and gates the solution with the
same `admit`, and copies every program it produced into one root. The comparison then
runs with `run_benchmark(frozen_library=...)`: every arm dispatches against that root,
nothing is written to it, and every row carries its one `library_hash`. The online mode,
where each arm grows its own library, remains for the cost curve.

Each of those seeds is an *occurrence*, and occurrences split by role on the
**instance identity** (`bench/splits.py`'s `occurrence_roles`, written onto every
ledger row; ADR-0005 decision 5):

| set | seeds | fault | variant occurrences (distinct instances) | replay occurrences |
| --- | --- | --- | --- | --- |
| smoke | 0, 1, 2, 4 | `diverged` | 4 | 0 |
| smoke | 0, 1, 2, 4 | `submodule_moved` | 4 | 0 |
| smoke | 0, 1, 2, 4 | `dirty_tree` | 4 | 0 |
| smoke | 0, 1, 2, 4 | `branch_renamed` | 4 | 0 |
| smoke | 0, 1, 2, 4 | `lockfile_conflict` | 3 | 1 |
| tune | 1000-1015 | `diverged` | 15 | 1 |
| tune | 1000-1015 | `submodule_moved` | 7 | 9 |
| tune | 1000-1015 | `dirty_tree` | 10 | 6 |
| tune | 1000-1015 | `branch_renamed` | 15 | 1 |
| tune | 1000-1015 | `lockfile_conflict` | 14 | 2 |
| eval | 2000-2039 | `diverged` | 32 | 8 |
| eval | 2000-2039 | `submodule_moved` | 15 | 25 |
| eval | 2000-2039 | `dirty_tree` | 14 | 26 |
| eval | 2000-2039 | `branch_renamed` | 30 | 10 |
| eval | 2000-2039 | `lockfile_conflict` | 28 | 12 |

**The independent observations are the distinct instances, and the counts are
measured, not inferred from the grid.** Each measurable fault now draws each state
shape's content per seed along declared Tier 1 axes — file bodies, local commit
count and subjects, and conflict hunk positions — so a set's variant occurrences
are exactly the distinct **instances** its seeds build. The per-resolution
breakdown the report carries (`bench.report.achieved_instances`) is:

| fault | resolution | instances in the eval set |
| --- | --- | ---: |
| `diverged` | `discard` | 18 |
| `diverged` | `merge` | 8 |
| `diverged` | `rebase` | 6 |
| `submodule_moved` | `init` | 5 |
| `submodule_moved` | `remove` | 5 |
| `submodule_moved` | `repin` | 5 |
| `dirty_tree` | `aside` | 4 |
| `dirty_tree` | `commit` | 4 |
| `dirty_tree` | `stash` | 6 |
| `branch_renamed` | `merge` | 9 |
| `branch_renamed` | `rename` | 12 |
| `branch_renamed` | `retrack` | 9 |
| `lockfile_conflict` | `keep_local` | 11 |
| `lockfile_conflict` | `take_upstream` | 17 |
| `config_renamed` | `drop_old` | 9 |
| `config_renamed` | `migrate_local` | 11 |
| `config_renamed` | `rename` | 7 |

The 40-seed eval set therefore builds **146 independent environments** -- 47 on the
first two faults, up from 6, 14 on `dirty_tree` since ADR-0027, 30 on `branch_renamed`
since ADR-0028, 28 on `lockfile_conflict` since ADR-0029 and 27 on `config_renamed`
since ADR-0041 -- and every resolution
reaches the owner's target of 4. `lockfile_conflict`'s versions follow from its drawn
package pair rather than being drawn on their own: drawn independently, no seed set
ever repeated an instance, and the cost curve is read from repeats. `dirty_tree/aside` sits exactly on it:
its only axis, the content flavour, has four values, so no seed set can draw a fifth.
The smoke (admit) set draws no `stash` state, so a library built on it holds no
`stash` program; a `stash` state accepts `commit` and `aside` too (ADR-0023), so a
fire of either there is correct. The counts come from
`FaultSpec.instance_for_seed` over the plan's seeds; reproduce the real-sandbox
measurement, including that distinct identities are distinct environments, with
`.venv/bin/python -m precondition_library.bench.instance_diversity`.

**The episode-level mismatch comparison is still underpowered, for a different
reason.** No interval could be attached at 6 environments. One arm's success
rate now carries a real interval: its 95% Wilson half-width near 0.5 is about
**±18.6pp** at the owner's target of 24, **±13.7pp** at the 47 the first two faults
build, **±12.2pp** at the 61 three build, **±10.1pp** at the 91 four build, **±8.8pp**
at the 119 the five git faults build and **±8.0pp** at the 146 all six build. But the
±10pp margin (item 10) is applied to the **difference** between two arms' rates, by
a TOST at a 90% interval, and that interval is wider. With `tost_equivalence` and
both arms at the same observed 50% rate — the most favourable case — the difference's
half-width is about **±22.5pp** at 24 per arm, **±16.5pp** at 47, **±14.6pp** at 61,
**±12.0pp** at 91, **±10.5pp** at 119 and **±9.5pp** at 146, and ±10pp first
becomes passable at **133 per arm** (so the six families' 146 clear it, in that most
favourable case only) (87 at an 80% rate, 53 at 90%; any observed
difference needs more). ADR-0006 keeps the margin at ±10pp, so at the achieved N the
expected verdict is "comparable", which the report already produces for an
underpowered run. The pre-registered primary
comparison is unaffected, because it is not computed from the episode loop: it is
the pair-level one over labelled (state, program) pairs (item 1), where the state
grid is crossed with seeds, no library accumulates between pairs, and the pair
count is chosen for power rather than for what a run can pay for (issue #5). The
episode loop's own comparative figure stays reported, labelled underpowered, with
its instance N shown.

**What more episodes would buy.** Replay occurrences for the cost curve, and
increasingly few new instances: the instance count is bounded by the declared draw
space and grows sublinearly with the seed count, so a larger seed set buys mostly
repeats. Independent observations are bought by more *draw space* — the held-out
axes issue #6 still owns (file names, submodule paths, branch names; Tier 2, which
need probe and body parameter binding) — or by the pair-level harness, which
crosses states with seeds and is not budget-bound.

The whole live sequence -- build, 2b tuning, frozen benchmark, admission factorial,
Figure 1 and the online arms -- is one command, `python -m
precondition_library.bench.live --api-key-env DEEPSEEK_API_KEY --out DIR`, which calls the
functions below in this order and writes every artifact under `DIR` (revision 68). Its
caller names the key's source; the package still finds none on its own. A single stage
is invoked like this. The ledger goes outside the repository; the API key
comes from the caller's environment and is never read by this repository
(`provider.py` reads no environment variables and no files):

```python
import os
import tempfile
from pathlib import Path

from precondition_library.bench import run
from precondition_library.bench.ledger import Arm
from precondition_library.bench.splits import TUNE_SEEDS
from precondition_library.provider import DeepSeekProvider
from precondition_library.tasks.registry import ambiguous_intents

provider = DeepSeekProvider(api_key=os.environ["DEEPSEEK_API_KEY"])
run.run_benchmark(
    arms=list(Arm),
    faults=[intent.fault for intent in ambiguous_intents()],
    occurrences=len(TUNE_SEEDS),
    seeds=list(TUNE_SEEDS),
    out=Path(tempfile.gettempdir()) / "precondition-ledger.jsonl",
    model="deepseek-flash",
    provider=provider,
)
```

---

## 8. Failure handling

Rule: **no silent retries, no invisible costs.** Every degradation is attributed
to an arm and recorded with a reason.

| event | action | recorded as |
| --- | --- | --- |
| replay misses postconditions | demote program; fall back to ReAct **for this episode only**; fallback tokens attributed to the arm | `misfired` (derived), with `outcome` showing how the episode then ended |
| a fired body names a declared parameter this environment cannot bind | no command runs; demote the program; fall back to ReAct **for this episode only** | `replay_failure_reason`, with `outcome` showing how the episode then ended (issue #76) |
| same program mismatches twice | withdraw from dispatch; retain for analysis | `quarantined` |
| compile fails admission | stored as a `candidate` and its rejection reason recorded, not discarded; episode keeps its token cost | `candidate`, `admitted=false`, `compile_failure_reason` |
| guard refuses the body | no execution; fall back to ReAct | `refusal` + `refusal_reason` |
| replay exceeds timeout | process killed, sandbox destroyed | `fail`, `timed_out` |
| no program applies | solve with LLM, compile, admit | `fallback` (expected, not a failure) |
| provider error / rate limit / transport failure (timeout, dropped connection) | capped backoff retry (429, 5xx and transport failures; never another 4xx or a malformed body) | `fail`, partial token spend retained, every attempt counted in `llm_calls` |
| sandbox build fails | episode invalidated, counted | `invalid` + `invalid_reason` |

A fallback is a **cost, not a non-event**: every time no program applies, the arm
pays full price. The cost curve is therefore partly a coverage story, which is
why `fallback` ("no program for this state") and `mismatch` ("a program claimed
there was one, wrongly") are distinct outcomes.

---

## 9. Safety and prompt injection

The compile step reads repository content — commit messages, file contents,
branch names, submodule URLs — all of which an attacker controls, and then
authors code that **runs unattended on later episodes**. This is a real
vulnerability, not a theoretical one, and it is designed against rather than
noted.

| threat | mitigation |
| --- | --- |
| Injected instructions in repo content steer the model while compiling | repo-derived text is passed in a delimited untrusted channel with explicit framing that it is data, never instruction |
| The compile step executes something while "testing" | the compile step **never executes** generated code; execution belongs to `runtime`, on a throwaway sandbox, behind the guard |
| Generated body exfiltrates repository contents | guard refuses outbound network calls and reads of credentials/SSH keys/env vars |
| Generated body writes outside its environment | guard refuses paths outside `env_root`; `HOME` is redirected into the sandbox and the environment is reduced to an allowlist (`PATH`, locale, `TMPDIR`, plus the pinned `GIT_*` variables), so an API key exported by the operator cannot reach the body |
| Generated body force-pushes to a shared remote | guard refuses pushes to any remote not recognised as a sandbox |
| Generated body undoes the git hardening per invocation -- `git -c core.hooksPath=<dir the attacker committed>` runs that directory's hooks, because command-line config wins over the pinned environment (verified, issue #159) | the guard refuses every per-invocation config override (`-c`, `--config-env`) outside an **allowlist** of one -- `protocol.file.allow=always`, which the submodule family needs -- and any body that sets `GIT_CONFIG_*` in its own environment; the ReAct `run_git` tool applies the same parser (`sandbox.disallowed_git_config_overrides`), and the injection suite measures the tier. A persistent repository `git config` write needs no refusal: the environment's `GIT_CONFIG_COUNT` outranks repository config |
| The agent's git tool has git run a command for it -- an alias set to `!cmd`, a filter or diff driver, `remote.<name>.uploadpack`, `submodule foreach`, `rebase --exec` (the fourth live smoke run's agent ran `sed` on the host through an alias) | the tool refuses those forms (`agents.command_screen`): config writes only to an allowlisted key and never to a `!` value, and no subcommand or option that runs a program. A screen over the arguments, not a boundary (#10) |
| Generated body is destructive *inside* the sandbox | **accepted** — that is what the sandbox is for; the sandbox is destroyed afterwards |
| A generated precondition or postcondition **changes** the state it is observing | probes are screened by the same guard as bodies, and every probe runs between two snapshots of the sandbox (`runtime.probes.sandbox_state`: working-tree files, index, `.git/config`, HEAD, and every ref of the clone and the upstream). A probe that changed any of them is refused, reported as not holding with `refused` set, and admission rejects a program whose precondition is refused rather than reading the refusal as a "no" (issue #10). Not covered: other files under `.git` (objects a probe adds are unreferenced; hooks are disabled by `core.hooksPath`), and anything outside the sandbox, which is the guard's job |
| A repository-derived value carries shell -- an upstream's default branch named `x;>PWNED` (git allows `;`, `$`, `(`, `>`, quotes in a branch name) or a `.gitmodules` path with metacharacters -- and `substitute` pastes it into a probe or body run under `bash -c` | `runtime.probes` binds a repository-derived value (`upstream_remote`, `upstream_branch`, `submodule_path`) only when it is letters, digits and `._/+-` with no leading `/` or `-` and no `..` component (`shell_safe`); anything else is left **unbound**, so a program that needs it is not applicable and nothing it carries runs. `work_dir` is the user's own choice and is not screened (ADR-0040) |

**Structural protections, not just screening:**

- Phase 1 never points a replay at a real repository. Admission, benchmarking,
  and replay all run on disposable sandboxes.
- The replay timeout bounds damage from a hung or looping body, and it bounds the
  **whole process group**: every body and probe runs in its own group (a new session
  on POSIX, a new process group on Windows), and on expiry the group is killed, so
  work a body put in the background cannot outlive it (`runtime.confine`, issue #10).
  On POSIX the command also runs with a per-file size limit (256 MiB) and core dumps
  off.
- `runtime/replay.py` cannot reach `provider` (enforced by test), so a replay
  cannot be talked into asking for help.
- A program may carry **argv-template `steps`** instead of a shell `body`
  (`runtime.templates`, issue #10): a closed catalogue of git operations the task
  family needs, each built into an **argv array** from the sandbox's bound values
  and the step's literals and run with no shell — so an attacker-controlled value
  (a commit message, a `.gitmodules` path) is one element git receives verbatim
  and cannot become code. Bound values are validated before they reach argv (refs
  against a charset, paths against a charset and realpath containment under the
  working tree), and the catalogue has no entry for the network, a force-push, or
  a write outside the tree. This is the positive boundary the free-form body
  lacks; the gold resolutions run through it. The model-compiled online arms still
  emit shell bodies behind the guard — migrating the compile step to emit `steps`
  is issue #4's territory and not done here.
- An adversarial injection suite (`bench.injection`, issue #10) drives the worst
  case these defences exist for: a program whose body or probe is attacker text
  verbatim, planted through each of the four origins above (commit message,
  filename, branch name, issue text), as if the compile step were fully steered.
  It reports containment per tier against planted canaries — a file outside the
  sandbox, a planted secret, a git hook, a backgrounded writer — and the network
  and force-push tiers, which cannot run without real egress or a real remote,
  are scored by the guard's refusal. The compile-time untrusted-channel tier
  needs a model and is reported as *not measured* rather than omitted. It does
  not run a model or touch a real repository.
- Each body and probe runs with **no network, where the host allows it**
  (`runtime.confine`, issue #10): a fresh, empty network namespace via
  `unshare --net`, a kernel boundary rather than the guard's textual screen. The
  sandbox's only remote is a local filesystem path, which needs no network, so
  isolation does not change a replay's result. The privilege to create a
  namespace may be withheld (a non-root runner, a locked-down container); then the
  command runs without it and `ReplayResult.network_isolated` records that it did —
  the owner's choice to run and record, not refuse. Linux only.

**Residual risk, accepted and documented:** a generated body may perform
destructive-but-permitted actions within its sandbox. Network isolation is
kernel-enforced where the host can create a network namespace (`unshare --net`,
above) and falls back — recorded as not isolated — to the textual and
environmental screen where it cannot: the guard refuses recognisable network
tools and URLs in the body, and the environment allowlist with a redirected
`HOME` keeps ambient credentials out, but neither stops an endpoint assembled at
run time (the guard's "What the guard cannot see") once the namespace is
unavailable. A separate unprivileged user and a real disk quota still need
privilege a process cannot give itself, so they are a **runner requirement** the
execution container must provide (ADR-0013, `docs/running-unattended.md`) rather
than library code; the library records whether the network namespace applied but
does not verify the user or the quota. A body that calls `setsid` leaves its process group and escapes the
group kill, and Windows gets the group kill but neither the resource limits nor
the network namespace. Running compiled programs against real repositories
requires human review of each program and is explicitly out of scope; the guard
is a seatbelt, not a sandbox boundary.

---

## 10. Testing strategy

```
GOLD FIRST      hand-written solutions must satisfy the fault checkers before
                any agent runs. A broken checker produces plausible numbers
                that mean nothing. INTENDED to be enforced by tests/ before
                anything else; NOT yet implemented -- test_checkers_against_gold
                is skipped until phase 1. Gold resolutions now exist for the two
                ambiguous intents (bench/gold/), but no checker has run against
                them.
DETERMINISM     same seed -> byte-identical faulted environment; the regression
                that would show up as variance between arms. Different seeds
                -> different instances when they draw different Tier 1 axis
                values, and the same instance only on a genuine repeat: measured
                over the 60 pre-registration seeds after ADR-0005, distinct
                `instance_for_seed` identities build distinct environments (issue
                #86), and `build_sandbox` refuses a draw whose observed resolution
                is not the declared one. Before ADR-0005, seeds selecting one
                resolution were byte-identical, commit SHAs included; they are not
                any more.
TEXT CONTROL    a bag-of-words classifier trained on the request text alone
                reports the informed/uninformed split. The uninformed regime
                cannot carry the resolution by construction (the sampler never
                consults state, so the AUC is 0.500 for any classifier and any
                phrasing list); LEAKAGE_CEILING gates it as a plumbing tripwire
                for state reaching the sampler, not as a leak detector, so it
                cannot certify the phrasing distribution. The wording is therefore
                checked directly instead: `test_no_phrasing_names_a_resolution`
                rejects any whole-word variant id in either channel, which is the
                literal leak the AUCs cannot see. A revealing paraphrase remains a
                review obligation. The informed regime
                is a genuine measurement: 0.962 / 0.945 -- the boundary
                condition where the wording nearly determines the resolution
                and the mechanism is not needed. Pinned by a positive control
                that fires on an intent whose informed wording fully determines
                the answer, so the control cannot pass by being a no-op. Only
                registered intents are measured; since ADR-0029 every fault
                has one, and `EXCLUDED_FROM_BENCHMARK` is empty.
IMPORT GRAPH    replay cannot reach provider, directly or transitively.
                IMPLEMENTED AND PASSING (tests/test_replay_isolated_from_provider.py)
ZERO TOKEN      stronger than the import test: replay runs with a provider whose
                complete() RAISES. Any hidden call path fails loudly at the
                integration level, where the import test only sees statically.
GUARD TABLE     bodies that must be refused, one line each to extend.
LEDGER INTEGRITY  every episode writes exactly one record; denominators must be
                reconstructable from the ledger alone.
NO SILENT SKIP  a skipped test must carry the issue that implements it. A test
                suite that quietly skips is a suite that passes for the wrong
                reason.
DECLARED STATE  every stub and every skipped test is declared in
                tests/test_declared_state.py and the declaration is asserted
                against the code, so implementing a stub or un-skipping a test
                fails CI until the declaration -- and the prose that restated it --
                moves too. A small claims table pins the status sentences that
                reduce to a mechanical fact, each failing with the document to
                update. It cannot detect a document contradicting another document,
                or one whose meaning contradicts the code; that stays a review
                obligation (CONTRIBUTING rule 5).
```

The zero-token test is the one that matters most **for the cost model**: the
replay invariant is what keeps the reported token totals honest, and the most
likely way to make them silently false is an import nobody noticed. Both a static
test and a runtime test guard it. It protects the cost model, not the primary
claim.

---

## 11. Phases

| phase | contents | exit criterion |
| --- | --- | --- |
| 0 | design, scaffold, invariants, CI | **done** — this document, committed skeleton |
| 1 | reframe after review; correct the experimental design (issues #3, #4, #9) | task text no longer labels the fault; arms share one frozen library and an identical admission gate; ground truth is decoupled from the artifact contract |
| 2 | dispatch-level benchmark harness, calibration sweep, power statement (#5, #6) | labelled (repo-state, candidate-program) pairs exist; arm 2 sweeps a threshold and arm 3 is one operating point (ADR-0008); a power statement is written |
| 3 | baselines that make the control arm credible (#7); safety hardening (#10) | the probe-classifier, intent-key, reflexion, and gold-script baselines run; the injection suite reports containment per defence tier |
| 4 | run the benchmark; publish the mismatch-vs-coverage curves | Figure 1 exists with Wilson intervals at each coverage point |
| 5 | episode loop as a demo, explicitly underpowered | the harness runs end to end; labelled underpowered wherever reported |
| 6 (optional) | DSH plugin wrapper around the admitted library | out of scope unless phases 1–5 land |

---

## 12. Decisions and rejected alternatives

| decision | choice | rejected alternative and why |
| --- | --- | --- |
| axis of contribution | the matched-coverage measurement (Claim 2), with the negative-sandbox admission criterion as a second candidate | **a novel architecture** — dropped (ADR-0001): compile-once replay is established prior art (PreAct, SkillDroid, Auto) and precondition dispatch is MACROPS, 1972 |
| headline claim | none — the project reports a measurement, not a claimed architecture | **compile-once, replay-without-model** — dropped as prior art, ADR-0001 |
| baseline | a ReAct-style observe/act/observe loop (Yao et al., ICLR 2023), given the same tools and sandbox | self-modifying agent — generations are serial and each needs a full evaluation sweep, so a short project sees too few generations to distinguish signal from noise |
| | | swarm of tiny agents — cheap per run but high variance, and a positive result is hard to attribute to any mechanism |
| | | memory-as-program — a real problem, but it needs long-horizon tasks that are themselves expensive to run |
| comparison | three arms (react / semantic / precondition) | two arms (react vs precondition) — cheaper, but then Claim 2 has no control and a reviewer is right to say semantic dispatch was never given a fair shot |
| dispatch mechanism | executable preconditions | text-similarity-only, lexical today with an embedding model behind the same seam as the intended replacement — it sees the state as words but cannot evaluate it, and is *hypothesized* to mis-fire on structurally different environments that read alike; testing that hypothesis is what the benchmark is for |
| | | one monolithic growing program with a dispatcher — the most striking demo, but it rots, and it cannot ablate *which* mechanism helped |
| **arm 2's baseline and seam (2026-09-20, ADR-0003; regime correction 2026-09-21 in revision 36, ADR-0004 — proposed)** | the shipped lexical scorer stays, and its baseline is quoted **per intent within the informed request regime** over the whole ambiguous subset, with strict top-1 operative and the AUC beside it — hand-written gold on the probe's seeds 0–3, **not** the pre-registered split; the uninformed regime is chance by construction and is reported beside it as a plumbing tripwire, never as baseline | **IDF/BM25 weighting** — measured on the informed regime: it raises the AUC (0.6380 Jaccard, 0.6658 cosine, against 0.6172) while *lowering* strict top-1 (13/24 and 13/24 against 15/24), and document frequencies need a corpus the one-method seam has nowhere to carry; **dropping the constant `intent` from `_program_text`** — rejected in ADR-0003, which records the then-pooled measurement; it would invalidate comparability with the recorded figures for no measured gain, and nothing in the regime correction revives it; **quoting one intent's figure as the family's** — within the informed regime the two ambiguous intents differ by more than 0.2 AUC (0.8125 against 0.6007), so neither describes the family; **pooling the informed and uninformed regimes** — measured: the pooled figure (0.5340 and 23/48) averages a channel where the wording nearly gives the answer away with one that is chance by construction and describes neither, so the probe now refuses to produce it (ADR-0004) |
| domain | branchy git maintenance chores | synthetic benchmark suite — cleanest ablation, but produces an agent nobody uses and leaves "does this matter" unanswered; chores were chosen because correctness is free to check and the recurrence is real |
| stack | standalone Python engine, own the loop | DSH plugin (TypeScript) — cannot ablate a loop the host owns; measuring loop cost from inside a tool is not possible |
| sandbox | disposable repos only | real forks — replayed programs execute unattended; real repositories are not an acceptable target for unreviewed generated code |
| benchmark episodes | manufactured by seeded fault injection | waiting for real faults to recur — 60 recurrences will not occur in three weeks, so the experiment could not reach N |
| **reframe after review (2026-09-13, ADR-0001)** | re-center on the open measurement: the dispatch comparison becomes the spine, the agent becomes its harness, Claim 1 is dropped as a contribution | **keep the agent central and merely rewrite the prose** — preserves the original vision, but the headline becomes "a modern instance of a classical mechanism" and invites "MACROPS for git chores"; the measurement is the only thing here that is actually open |
| | | **abandon the project** — the narrow gap is real and unmeasured and the engineering is already scaffolded; abandonment would be a reaction to losing a claim, not to losing the question |
| primary metric location | dispatch-level benchmark at matched coverage | episode-level paired mismatch difference — degenerate as designed: one fixed task sentence per fault made the task text the ground-truth label, so the control arm could not mis-fire by construction |
| unit of analysis (2026-09-27, ADR-0005) | the **environment** — a resolution plus its drawn Tier 1 axis values, the `instance_for_seed` identity — with a variant occurrence the first sight of each instance | treating each episode as an independent sample — pseudo-replication that inflates N and leaks the admitted instance into evaluation; and keying on the resolution (the rule before ADR-0005), which counted one environment many times because same-resolution seeds were byte-identical |
| citation policy | every citation checked against the source by the author before publication — a hygiene pass, not independent review | trusting a research agent's prior-art sweep — it produced five wrong attributions and one false negative claim about a repository in the author's own ecosystem |

### Open risks

1. **Arm 2 may saturate, or may be too weak to rank at all.** The arm scores
   lexical overlap today, so its ceiling is surface form; whether that ceiling is
   too low to rank the library is for the tune set to show, and a control arm
   that scores almost nothing would flatter preconditions. A real embedding model
   behind the same seam is the intended replacement, and a strong one may make
   similarity dispatch nearly as good as preconditions on five well-separated
   faults. If so, Claim 2 dies — and the design should add faults that are
   *phrased alike but structurally different*, which is where embeddings should
   fail and probes should not. Watch for this in phase 2, before the full run.

   The discrimination probe gives an exploratory reading of this risk before any
   episode runs (ADR-0003, narrowed by ADR-0004; gold programs, the declared
   state grid, seeds 0–3 — the probe is **not** the pre-registered plan, and
   these are not tune-set figures; re-measured per request regime in revision
   36). On the **informed** channel, where the wording nearly gives the
   resolution away, the shipped scorer scores **AUC 0.6172, strict top-1 15/24
   (63%)** against a 33% chance for three candidates, and that baseline hides the
   spread that is the risk made concrete — **0.8125 / 67%** on
   `restore_submodule_state` against **0.6007 / 58%** on
   `sync_fork_with_upstream`. The arm is above chance on both, so the lexical
   proxy is a competent baseline rather than a straw man and a win for
   precondition dispatch has to be won against it. The **uninformed** channel is
   chance by construction (**AUC 0.5000, strict top-1 8/24, 33%**) and is the
   plumbing tripwire, not baseline material: the 8 negative pairs belong to it,
   and a rise there would mean state had begun reaching the sampler. This is why
   ADR-0003 requires the per-intent breakdown beside any pooled figure, why
   ADR-0004 makes the informed regime the baseline and forbids pooling the two
   channels, and why the embedding swap stays open as the one remaining
   candidate: filling the seam would change the mechanism rather than the
   weighting, and on the informed regime the embedding does not improve on the
   lexical top-1 (**12/24** against **15/24**), so nothing measured so far does.
2. **Compile cost may dominate.** If compiling costs as much as several ReAct
   episodes, amortization needs many recurrences. The crossover point is the
   honest result — but it should be checked early, since it determines whether
   three occurrences is enough structure.
3. **Fault leakage.** Injected faults may leave detectable artifacts (a
   suspiciously named branch, a telltale commit message) that make recognition
   artificially easy. Two halves, and only one is checked: the *wording* half is
   gated by `test_no_phrasing_names_a_resolution`, while the *injected artifact*
   half needs a repo to inspect and so waits on the live sandbox (issue #4).
4. **Five faults may be too few** for a mismatch comparison with usable
   intervals. The extension rule in §7 covers it, at the cost of episodes.
5. **Model drift.** Provider-side model updates mid-experiment would confound
   everything. `model` is recorded per episode; a version change invalidates the
   affected run and requires re-running that arm.
6. **Every fault is converted (closed by ADR-0029).** `dirty_tree`, `branch_renamed`
   and `lockfile_conflict` were converted by #189, #190 and #191 and registered by
   ADR-0027 to ADR-0029, so no fault's request is a fixed sentence any more. The
   guard stays: a fault added later with a fixed sentence **must not be included in
   any dispatch measurement** until it gains an `IntentSpec` with two or more
   state-decided resolutions. Enforced rather than merely stated:
   `EXCLUDED_FROM_BENCHMARK` in `tasks/registry.py` would name it, and
   a test asserts every fault is either intent-covered or listed there, so a fault
   cannot be silently absent from both.

---

## 13. Prior work and positioning

Filled in 2026-09-13. **Every citation below was checked against the source by the
author** — title, authors, venue, and each attributed figure — after the sweep that
produced them was found to contain five wrong attributions and one false negative
claim about a repository in the author's own ecosystem. Nothing enters this
section, or the README, unchecked. The full table, with per-work caveats, is in
the README.

**This is not independent review.** The check was performed by the same author, so
treat it as a hygiene pass rather than external validation.

### What prior art settles

Compile-once replay is established, not novel. PreAct (arXiv:2606.17929) compiles
a computer-use trace into a state machine with per-state verification predicates
and parameter lifting, replaying 8.5-13× faster with no per-step LLM calls, and
documents a `cov=100%/score=0` "lossy replay" failure. SkillDroid
(arXiv:2604.14872) compiles GUI trajectories into parameterized templates and
replays with zero LLM calls. Auto (arXiv:2607.04542) compiles
witnessed-deterministic spans into signed artifacts gated on differential replay,
reporting 2,775 vs 17,692 µ$ over 300 items and 48.9% silently wrong under a loose
guard. LATM (arXiv:2305.17126, ICLR 2024) builds a reusable tool from
demonstrations, then has a cheaper model invoke it — note that its replay phase
still calls an LLM, so it evidences *cheaper* replay, not free replay.

Executable-precondition dispatch is classical, not novel. MACROPS (Fikes, Hart &
Nilsson, *Artificial Intelligence* 3(4):251-288, **1972**) dispatches a cached
generalized plan by testing precondition kernels against live state, with an
explicit replan fallback; 1971 is the separate STRIPS paper. Soar chunking (Laird,
Rosenbloom & Newell, *Machine Learning* 1(1):11-46, 1986) caches compiled rules
re-fired on matching conditions. Options (Sutton, Precup & Singh, *Artificial
Intelligence* 112(1-2):181-211, 1999) give an option's initiation set the role a
precondition plays for a policy — an analogy, not an identity, since an option
also carries a termination condition that this project's preconditions lack.
Adaptation-guided retrieval (Smyth & Keane, *Artificial Intelligence*
102(2):249-293, 1998) argues it is often unwarranted to assume that the most
similar case is also the most appropriate for reuse.

### What brackets the open question without settling it

A deployed deterministic executability gate (arXiv:2608.01050) removes 59.4% of
matched skill-message pairs and 59.1% of skill-description tokens, with 90.5%
context reduction, and its gate-removed counterfactual selected a skill blocked as
non-executable in 7.8% of conversations. But it is a cascade *after* semantic
recall — it can only prune, never rescue a recall miss — it covers ten skills in
one domain family, and no head-to-head comparison is reported.

A retrieval-risk benchmark (arXiv:2606.10388v2) measures top-K exposure of harmful
sibling skills at HSR@3 0.346-0.372 for public retrievers versus 0.007 for a
controlled resolver. That is retrieval *exposure*, not wrong-program *execution*,
and again no precedence-versus-similarity dispatch comparison. These figures
belong to v2 under its v2 benchmark name; the earlier title has different, much
smaller figures.

### The hypothesis that survives

**The hypothesis:** at matched dispatch coverage, executable-precondition dispatch
will mis-fire less often than text-similarity dispatch, where mismatch is scored as
an outcome orthogonal to episode success. The mechanism is 1972; **that comparison
has not been measured.** Running it is the intended contribution, and no result
exists yet — this section describes a hypothesis and an intended output, not a
finding.

The negative-sandbox admission criterion (§6) is the secondary candidate: a
requirement that no confirmed work publishes, with the closest published gate
being exit-condition-based and applied as a cascade after semantic recall — a
different test. Its cheapness and enforceability are design intentions, not
findings.

---

## 14. Interaction with the operator-supplied prompt

A system prompt and tool surface for the agent will be supplied separately and
is expected to change during development. The design isolates it:

```
agents/react.py    system prompt + tool schemas   <- swappable
agents/compile.py  compile prompt                 <- swappable
        ────────────────────────────────────────
        dispatch, admission, runtime, ledger      <- NOT swappable per-arm
```

Changing the prompt must never change what the experiment measures. If a change
to the prompt alters arms 2 and 3 differently, it has stopped being a prompt
change and become an architecture change, and must be recorded as such.