{ "meta": { "title": "Bug Hunt Bench", "subtitle": "105 real bugs hidden in two production repos. Frontier coding models find and fix what they can, graded blind.", "planted_total": 105, "repo1_total": 45, "repo2_total": 60, "survivors": 26, "models": 46, "bug_count": 105, "repo1_bug_count": 45, "rows": 185, "runs": 227, "updated": "2026-10-14", "source_repo": "https://github.com/phuryn/bug-hunt-bench/tree/main/results", "repo_url": "https://github.com/phuryn/bug-hunt-bench", "author": "Pawel Huryn", "author_url": "https://www.productcompass.pm", "author_x": "https://x.com/PawelHuryn" }, "method": [ "Two unrelated production codebases: a ~28K-line TypeScript VS Code extension (45 bugs) and a ~60K-line React/Supabase LMS (60 bugs). Most bugs are real regressions reverted from the repos' own fix history, each with its fix commit.", "The test suites stay green, so nothing points at the answers. Every model gets the identical prompt, one round per repo, in its own native agentic CLI.", "Grading is blind: the model's diff is the ground truth, not its report. An independent judge scores against a withheld answer key and never learns which model produced the diff.", "The score counts planted bugs only. Some models bug-max - they report a long list of real but irrelevant problems, and that list has no answer key and no ceiling, so counting it would just reward volume. Unplanted fixes are tracked separately and never scored." ], "glossary": { "fixed": "Planted bugs fixed and verified against the withheld answer key. The only number in the score.", "partial": "The defect was addressed but not fully closed. Not counted in the score.", "claimed_only": "Named in the model's report but not fixed in its diff - including bugs it saw and chose not to touch. Not counted in the score.", "extras": "A real defect the model fixed that nobody planted. Genuine work - but the set has no answer key and no ceiling, so it cannot be scored reliably. Counted, never added to the score.", "cost_bill": "An actual invoice or credits delta.", "cost_list": "A token-count estimate at published list rates, not a bill.", "cost_floor": "Reconstructed lower bound - this CLI reports context occupancy, not billing.", "cost_free": "Served free at the time of the run.", "effort_verified_ceiling": "This model's highest reasoning setting. We checked that changing the setting actually changes what the model does.", "effort_verified": "We checked that changing this reasoning setting actually changes what the model does.", "effort_first_party": "A setting the vendor documents in their own tool. Taken at face value, not separately tested.", "effort_clamped": "We asked for a higher setting and the tool quietly ran a lower one. Published as a correction.", "effort_inert_default": "This route ignores the effort setting entirely, so the model ran at its own default. There was no working dial to turn.", "effort_default": "No reasoning setting was asked for, because this route offers none to ask for. Unlike an ignored dial, nothing was tested here - this says what we requested, not what the route does with it.", "turns": "Model steps in a run: each tool call (a shell command, a file read or edit, a search) plus the final answer, summed over both repos, averaged across runs on a mean row. It is the count Claude Code reports as its own turn count, taken from every run's own log the same way." }, "caveats": [ "52 of the 116 rows on the board are means of two or more runs (22 of the 26 in the default view); the rest are single runs. 40 configurations have now been run three times at identical settings - same model, prompt, harness, route, effort flag and judge - and their three-run ranges are 1, 1, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 7, 7, 7, 8, 8, 8, 9, 9, 10, 10, 13, 14 points of 105. The widest spread was 43, 54 and 57; the tightest was 31, 32 and 32. Rows on this board routinely sit within a point or two of each other, so a small gap between two single runs may be nothing at all. WHAT THIS DOES NOT SAY: which settings are steady. Spread does not fall cleanly as effort rises - Gemini 4 Argon's rungs came out 6, 13 and 8 going up, Grok 4.7's rungs came out 10, 7, 5 and 6 going up, Muse Spark 1.3's rungs came out 9, 6, 3, 9 and 2 going up, Opus 5.5's rungs came out 6, 1, 1, 2 and 4 going up, Sonnet 5.5's rungs came out 3, 10, 5, 5 and 14 going up - and a range drawn from three runs is a very noisy estimate of spread. 12 models have been measured three times at max effort so far. Treat every one of these as a spread that has been SEEN, not a spread that has been bounded.", "Wall clock is the same measure on every row: repo 1 plus repo 2 agent time, with dependency install and health checks excluded. It is taken on a working machine, not an isolated one - other agents and applications routinely run alongside - so read small differences in time loosely; scores, tokens and cost are unaffected. Which runs overlapped is recorded per leg in the receipts on GitHub. One figure is adjusted: the retired-Gemini-CLI run excludes a 35-minute harness stall, the only number here that is not what the harness measured.", "Cost is not one kind of number: some figures are an actual bill, some are a token-count estimate at published list rates, one CLI only allows a reconstructed lower bound, and one run was free. The key under the board names which is which - they are not interchangeable, and none of them is a quote for what you would pay.", "26 of the 105 bugs have survived every model in every run to date. The count moves as new runs land: it fell from 49 to 44 during the 27-28 August waves.", "Turns count model steps: every tool call - a shell command, a file read or edit, a search - plus the final answer, summed over both repos. That is exactly what Claude Code reports as its own turn count, and it is the one definition every harness here can supply, so each run is counted from its own log the same way: Codex CLI, which counts a whole run as a single turn, by its commands and edits; Grok by the tool calls in its session log. A turn is a step, not a unit of work - one command can do several reads, and harnesses differ in how much a single call does - so turns compare most cleanly within one harness. 5 current rows have no turn count because their logs were not kept, and are left off that map rather than estimated." ], "variance_short": "52 of the 116 rows are means of two or more runs (22 of the 26 in the default view); the rest are single runs. Running a configuration three times at identical settings moved its score from the first run by a median 1.7 points of 105 - by more than 2 in 18 of 40 cases, by more than 5 in 3 - and up about as often as down (21 up, 18 down). Read a gap of a few points between two single runs as possible noise.", "runs": [ { "id": "Sonnet 5.5 (max effort)", "n_runs": 1, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 57, "repo1_fixed": 24, "repo2_fixed": 33, "partial": 3, "claimed_only": 2, "extras": 28, "false_positive_fixes": 0, "wall_min": 257.8, "cost_usd": 154.05, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-28 by the mean of 2 (55.5 of 105). This row reported 57 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 54 to 57. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat this as one draw until the two replicates land. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at max effort through Claude Code on the first-party API, measured the evening it shipped. 57 of 105: 24 of 45 on repo 1 and 33 of 60 on repo 2, 28 genuine extras, 257.8 minutes, $154.05. ONE RUN so far; two replicates follow and will supersede this row. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** IT KEEPS WORKING AFTER IT WRITES ITS REPORT *** On repo 1 it took 915 turns (the harness's own count); it first wrote its report 66 minutes in and its last model call came at minute 196 - 129 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 651 turns (the harness's own count); it first wrote its report 57 minutes in and its last model call came at minute 62 - 5 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. Its own repo-1 report says each fix carries a regression test that it confirmed fails with the fix reverted - a mutation check - which is where the extra turns went. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 1,001,140 tokens; 3 compactions across 2 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge returned an empty verdict set once and was re-run - the re-run is the scored result. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 36, 37, 38, 39, 40, 42, 43, 44, 45, 47, 49, 50, 51, 52, 55, 56, 59, 60, 61, 62, 64, 69, 70, 71, 73, 76, 77, 78, 79, 80, 83, 84, 91, 92, 94, 95, 96, 97, 98, 99, 100, 102 ], "turns": 1566, "turns_repo1": 915, "turns_repo2": 651, "group_rank": 500 }, { "id": "Sonnet 5.5 (max effort) - mean of 2", "n_runs": 2, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 55.5, "repo1_fixed": 22.5, "repo2_fixed": 33, "partial": 3, "claimed_only": 1, "extras": 24.5, "false_positive_fixes": 0, "wall_min": 234.7, "cost_usd": 134.79, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-29 by the mean of 3 (51.3 of 105). This row reported 55.5 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 43 to 57. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 57 and 54 of 105 - spread 3. Read the mean with that width, and note that two runs bound variance poorly - the third is still to come. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at max effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 57 and 54 of 105 (repo 1: 24, 21 - repo 2: 33, 33). Mean 55.5, range 3. *** THIS SUPERSEDES SONNET 5.5 (MAX EFFORT) (57 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** WALL AND COST *** walls 257.8, 211.7 minutes, mean 234.8; cost $154.05, $115.55, mean $134.80. *** GENUINE EXTRAS *** 28 and 21 across the runs, mean 24.5, never folded into the score. *** IT KEEPS WORKING AFTER IT WRITES ITS REPORT *** On repo 1 it took 915, 539 turns (the harness's own count); it first wrote its report 66, 58 minutes in and its last model call came at minute 196, 113 - 129, 55 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 651, 554 turns (the harness's own count); it first wrote its report 57, 49 minutes in and its last model call came at minute 62, 98 - 5, 49 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. Its own repo-1 report says each fix carries a regression test that it confirmed fails with the fix reverted - a mutation check - which is where the extra turns went. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 1,003,561 tokens; 5 compactions across 4 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge on run 1 returned an empty verdict set once and was re-run - the re-run is the scored result. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 36, 37, 38, 39, 40, 42, 43, 44, 45, 47, 49, 50, 51, 52, 53, 55, 56, 59, 60, 61, 62, 64, 66, 69, 70, 71, 73, 76, 77, 78, 79, 80, 82, 83, 84, 91, 92, 94, 95, 96, 97, 98, 99, 100, 102, 105 ], "bug_hits": { "13": 2, "16": 2, "17": 2, "20": 2, "21": 2, "22": 2, "23": 2, "25": 2, "26": 1, "27": 2, "28": 2, "29": 2, "30": 2, "32": 1, "33": 2, "34": 2, "36": 2, "37": 1, "38": 1, "39": 2, "40": 2, "42": 1, "43": 2, "44": 2, "45": 2, "47": 2, "49": 2, "50": 2, "51": 2, "52": 1, "53": 1, "55": 2, "56": 2, "59": 2, "60": 2, "61": 2, "62": 1, "64": 2, "66": 1, "69": 2, "70": 2, "71": 2, "73": 1, "76": 2, "77": 2, "78": 2, "79": 2, "80": 2, "82": 1, "83": 2, "84": 2, "91": 2, "92": 2, "94": 2, "95": 2, "96": 2, "97": 2, "98": 2, "99": 2, "100": 2, "102": 1, "105": 1 }, "coverage_runs": 2, "turns": 1330, "turns_repo1": 727, "turns_repo2": 602, "group_rank": 500 }, { "id": "Sonnet 5.5 (max effort) - mean of 3", "n_runs": 3, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-29", "color": "#E68F74", "fixed": 51.3, "repo1_fixed": 22.7, "repo2_fixed": 28.7, "partial": 3, "claimed_only": 1, "extras": 26.3, "false_positive_fixes": 0, "wall_min": 287.1, "cost_usd": 154.03, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 57, 54 and 43 of 105 - spread 14. Read the mean with that width. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at max effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 57, 54 and 43 of 105 (repo 1: 24, 21, 23 - repo 2: 33, 33, 20). Mean 51.3, range 14. *** THIS SUPERSEDES SONNET 5.5 (MAX EFFORT) - MEAN OF 2 (55.5 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** WALL AND COST *** walls 257.8, 211.7, 392.0 minutes, mean 287.2; cost $154.05, $115.55, $192.52, mean $154.04. *** GENUINE EXTRAS *** 28, 21 and 30 across the runs, mean 26.3, never folded into the score. *** IT KEEPS WORKING AFTER IT WRITES ITS REPORT *** On repo 1 it took 915, 539, 1145 turns (the harness's own count); it first wrote its report 66, 58, 64 minutes in and its last model call came at minute 196, 113, 332 - 129, 55, 268 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 651, 554, 688 turns (the harness's own count); it first wrote its report 57, 49, 56 minutes in and its last model call came at minute 62, 98, 60 - 5, 49, 4 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. Its own repo-1 report says each fix carries a regression test that it confirmed fails with the fix reverted - a mutation check - which is where the extra turns went. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 1,003,561 tokens; 8 compactions across 6 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge on run 1 returned an empty verdict set once and was re-run - the re-run is the scored result. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 36, 37, 38, 39, 40, 42, 43, 44, 45, 47, 49, 50, 51, 52, 53, 55, 56, 59, 60, 61, 62, 64, 66, 68, 69, 70, 71, 73, 74, 76, 77, 78, 79, 80, 82, 83, 84, 91, 92, 94, 95, 96, 97, 98, 99, 100, 102, 105 ], "bug_hits": { "13": 3, "16": 3, "17": 3, "20": 3, "21": 3, "22": 3, "23": 3, "25": 3, "26": 2, "27": 3, "28": 3, "29": 3, "30": 3, "32": 2, "33": 3, "34": 3, "36": 2, "37": 1, "38": 2, "39": 3, "40": 3, "42": 2, "43": 3, "44": 3, "45": 3, "47": 3, "49": 3, "50": 3, "51": 2, "52": 1, "53": 1, "55": 2, "56": 2, "59": 2, "60": 2, "61": 3, "62": 1, "64": 2, "66": 1, "68": 1, "69": 2, "70": 3, "71": 3, "73": 1, "74": 1, "76": 3, "77": 3, "78": 2, "79": 2, "80": 3, "82": 2, "83": 3, "84": 3, "91": 3, "92": 3, "94": 2, "95": 2, "96": 2, "97": 3, "98": 3, "99": 3, "100": 3, "102": 1, "105": 1 }, "coverage_runs": 3, "turns": 1497, "turns_repo1": 866, "turns_repo2": 631, "group_rank": 500 }, { "id": "GPT-6 Astra (max effort)", "n_runs": 1, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-04", "color": "#1B9E77", "fixed": 48, "repo1_fixed": 24, "repo2_fixed": 24, "partial": 0, "claimed_only": 0, "extras": 45, "false_positive_fixes": 0, "wall_min": 78.8, "cost_usd": 31.21, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same setting (45.0). This row is one of three runs that scored 48, 40 and 47, and it is the HIGHEST of the three - which is the reason the replication was worth buying rather than a fault in this run. Nothing here is corrected by arithmetic and the receipt stands; what does not survive is using 48 as the board's ceiling. Read the mean row's range - 8 points - before quoting any margin that rests on this number.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 60, 61, 62, 66, 68, 69, 71, 74, 76, 77, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": 221, "turns_repo1": 94, "turns_repo2": 127, "group_rank": 0 }, { "id": "GPT-6 Astra (max effort) - mean of 3", "n_runs": 3, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-14", "color": "#1B9E77", "fixed": 45, "repo1_fixed": 22, "repo2_fixed": 23, "partial": 1, "claimed_only": 0, "extras": 55, "false_positive_fixes": 0, "wall_min": 89.8, "cost_usd": 33.03, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Every figure on this row is a MEAN of three runs, carried to one decimal - 45.0 of 105. The three runs scored 48, 40 and 47: a range of 8, which is the widest spread measured on any row above 30 here and is wider than the gap between this row and the next. This row SUPERSEDES the single-run max row (48/105) published 2026-09-04, which was the best of the three; the supersession is about method, not a correction, and that receipt stands. READ THE TOP OF THIS BOARD ACCORDINGLY: the leader's own spread is larger than the distance to the rows beneath it, so the ordering at the top is not something three runs can establish. The rows it is being compared against are still single runs with spreads nobody has measured.", "note": "THE TOP ROW OF THIS BOARD, RUN THREE TIMES. Runs scored 48, 40 and 47 of 105 (repo 1: 24, 21, 21 - repo 2: 24, 19, 26). Mean 45.0, range 8. *** WHY THIS ARM MATTERED MOST *** Every other row is read against the leader, so the leader was the single run whose spread was worth the most and known the least. It is still the leader: 45.0 is the highest figure on the board and no other row is close to it at n=3. What changed is the width. *** THE RANGE IS 8 POINTS, AND IT IS NOT A SMALL NUMBER HERE *** Eight points of 105 is wider than the gap between the leader and the row below it, wider than most adjacent pairs anywhere on this board, and it sits on a configuration nothing about which was varied: same model, prompt, harness, route, effort flag and judge. The single run published on 2026-09-04 scored 48, which is the TOP of its own three-run range, so every comparison drawn against 48 was drawn against this configuration's best day rather than its typical one. That is not a correction to the old row - 48 happened, and its receipt stands - it is a correction to what the number could be used for. *** WHAT IT DOES AND DOES NOT MOVE *** It does not change the ORDER at the top: 45.0 still leads, and the gap to the next row is now 2 points rather than 5. It does mean that gap is inside the leader's own measured spread, so 'Astra max beats the row below it' is no longer something this board can assert - the row below is itself a single run with an unmeasured spread. Two numbers of unknown width two points apart support no ordering at all. *** THE UNPLANTED COLUMN MOVED FAR MORE THAN THE SCORE *** Genuine extras came in at 45, 58 and 62 across the three runs, a range of 17 on a count that is not scored. The score varied by 8 while the unscored work varied by 17, which is worth knowing before anyone reads a single run's extras figure as a property of a model. All six legs were clean: every leg exited 0, none carries a handicap note, and zero claimed-only fixes were recorded across all six - this arm claimed nothing the judge did not confirm, on any run. Per-leg receipts (repo 1 / repo 2): wall 37.2/41.6, 51.3/44.7 and 35.9/58.7 minutes; cost 15.07/16.14, 17.59/16.14 and 12.33/21.82 dollars. Run 3 ran 20 per cent longer than run 1 and scored one point lower. *** ONE JUDGING NOTE, BECAUSE IT NEARLY BECAME A SILENT DIFFERENCE *** Run 3's repo-2 leg created a scratch directory for itself, holding 300 copies of the files it was about to edit. Every copy was verified byte-identical to the pristine repo, so it contained no fix and no evidence, but left in the diff it would have added two megabytes to that leg's judge packet - a packet neither of the other two runs had. It is excluded, so all three runs were judged on the same shape of evidence. The alternative repair, raising the size limit, would have judged one replicate on a materially different packet from its own siblings. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. billing=LIST ESTIMATE, NOT A BILL: token counts at published list rates.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 56, 59, 60, 61, 62, 66, 68, 69, 70, 71, 74, 76, 77, 78, 79, 80, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 3, "22": 3, "23": 3, "25": 3, "26": 2, "27": 2, "28": 2, "29": 3, "30": 3, "33": 3, "34": 3, "35": 2, "36": 3, "37": 3, "39": 3, "40": 3, "41": 3, "42": 1, "43": 3, "44": 3, "45": 3, "46": 1, "47": 3, "49": 3, "50": 3, "51": 2, "56": 1, "59": 1, "60": 1, "61": 1, "62": 3, "66": 1, "68": 3, "69": 2, "70": 1, "71": 3, "74": 2, "76": 3, "77": 3, "78": 1, "79": 2, "80": 1, "82": 3, "83": 3, "84": 3, "92": 3, "95": 3, "97": 1, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 222, "turns_repo1": 104, "turns_repo2": 118, "group_rank": 0 }, { "id": "GPT-6.1 Sol (max effort) - mean of 3", "n_runs": 3, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-30", "color": "#2BA580", "fixed": 44.3, "repo1_fixed": 22.3, "repo2_fixed": 22, "partial": 3, "claimed_only": 0.3, "extras": 65.3, "false_positive_fixes": 0, "wall_min": 139.3, "cost_usd": 5.88, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 44, 42, 47 of 105 - spread 5. Read the mean with that width. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Max is not this model's top tier (the account also offers ultra). Cost is a list-rate estimate; no request reached the long-context rates. Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at max effort through OpenAI's own Codex CLI, on a ChatGPT account, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 44, 42 and 47 of 105 (repo 1: 21, 21, 25 - repo 2: 23, 21, 22). Mean 44.3, range 5. *** THIS SUPERSEDES GPT-6.1 SOL (MAX EFFORT) (44 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, pinned CLI, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. *** WALL, COST AND TURNS *** walls 112.4, 175.1, 130.5 minutes; cost $6.56, $6.16, $4.92; turns 280, 294, 236. *** GENUINE EXTRAS *** 59, 58, 79 across the runs, never folded into the score. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at low 31, GPT-6.1 Sol at medium 28, GPT-6.1 Sol at high 38, GPT-6.1 Sol at xhigh 41. At the same tier the board publishes GPT-6 Sol at max 29.3 (mean of 3), GPT-6 Astra at max 45 (mean of 3), Sonnet 5.5 at max 51.3 (mean of 3) of 105. *** MAX IS NOT THIS MODEL'S TOP TIER *** the account's own Codex model list offers low, medium, high, xhigh, max, ultra for gpt-6.1-sol - 'ultra' sits above max ('Maximum reasoning with automatic task delegation'), so this row is not marked as the model's ceiling. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at max. *** AN EARLIER ATTEMPT IS VOIDED IN THE LEDGER *** Codex's sandbox was misconfigured on the machine and refused every command the model tried, so it never ran one; the runner marked it void (exit 90) and the published figures are clean re-runs. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 56, 60, 62, 66, 68, 69, 70, 71, 73, 74, 76, 77, 79, 80, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "17": 1, "20": 3, "21": 2, "22": 3, "23": 3, "25": 2, "26": 2, "27": 3, "28": 3, "29": 3, "30": 3, "32": 1, "33": 3, "34": 3, "35": 3, "36": 1, "37": 3, "39": 3, "40": 3, "41": 3, "42": 1, "43": 3, "44": 3, "45": 3, "46": 2, "47": 3, "49": 3, "50": 3, "51": 3, "56": 1, "60": 1, "62": 3, "66": 1, "68": 2, "69": 3, "70": 1, "71": 3, "73": 1, "74": 1, "76": 3, "77": 3, "79": 1, "80": 2, "82": 3, "83": 3, "84": 2, "92": 3, "95": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 270, "turns_repo1": 119, "turns_repo2": 151, "group_rank": 100 }, { "id": "GPT-6.1 Sol (max effort)", "n_runs": 1, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-29", "color": "#2BA580", "fixed": 44, "repo1_fixed": 21, "repo2_fixed": 23, "partial": 3, "claimed_only": 0, "extras": 59, "false_positive_fixes": 0, "wall_min": 112.4, "cost_usd": 6.56, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 3 (44.3 of 105). This row reported 44 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 42 to 47. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat 44 as one draw. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model; the harness is not held fixed across that comparison. Max is not this model's top tier (the account also offers ultra). Cost is a list-rate estimate; no request reached the long-context rates (Codex caps the context for this model below 272K). The wall figure may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 44 of 105: 21 of 45 on repo 1 and 23 of 60 on repo 2, 59 genuine extras, 112.4 minutes, $6.56, 280 turns (133 and 147). ONE RUN, measured the night the model became reachable. For comparison the board publishes GPT-6 Sol at max 29.3 (mean of 3), GPT-6 Astra at max 45 (mean of 3), GPT-5.6 Sol at max 43.5 (mean of 2), Sonnet 5.5 at max 51.3 (mean of 3) of 105. *** MAX IS NOT THIS MODEL'S TOP TIER *** the account's own Codex model list offers low, medium, high, xhigh, max, ultra for gpt-6.1-sol - 'ultra' sits above max ('Maximum reasoning with automatic task delegation'), so this row is not marked as the model's ceiling. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** AN EARLIER ATTEMPT IS VOIDED IN THE LEDGER *** Codex's sandbox was misconfigured on the machine and refused every command the model tried, so it never ran one; the runner marked it void (exit 90) and this row is a clean re-run. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million - half GPT-6 Sol's cached rate. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped either leg - the only load this board's marker records - but the laptop was running several interactive coding-agent sessions that evening, at times near full CPU, so the wall figure may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 35, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 62, 66, 68, 69, 71, 73, 74, 76, 77, 80, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": 280, "turns_repo1": 133, "turns_repo2": 147, "group_rank": 100 }, { "id": "Muse Wonder Full 1.0 (max effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Wonder Full 1.0", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "first_party", "date": "2026-10-13", "color": "#D6336C", "fixed": 44, "repo1_fixed": 22, "repo2_fixed": 22, "partial": 2, "claimed_only": 0, "extras": 53, "false_positive_fixes": 0, "wall_min": 138.2, "cost_usd": null, "cost_kind": "unknown", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-10-14 by the mean of two runs (42.0 of 105). This row reported 44 of 105 from a single run. Nothing here is retracted - it is the receipt for what the board showed before run 2.", "caveat": "One run (44 of 105). Cost unknown: Meta publishes no price for this model. Effort max is the top tier Meta lists for it, not magnitude-probed. Both repos ran at the same time.", "note": "Meta's Muse Wonder Full 1.0 at max effort through Muse Code, Meta's own agent CLI, released 2026-10-12 alongside Muse Wonder 1.0. ONE RUN: 44 of 105 - 22 of 45 on repo 1 and 22 of 60 on repo 2 - with 2 partial fix(es), 0 claimed-only and 53 genuine unplanted extras; 138.2 minutes. *** AGAINST MUSE WONDER 1.0 *** same harness, route and effort: Wonder's mean of three is 31.7 (30, 36 and 29), 12.0 on repo 1 and 19.7 on repo 2. Full's lead is almost all on repo 1 (22 against 12.0); on repo 2 it sits near the top of Wonder's range. Muse Spark 1.3 at max stands at 32.2. A single run: Wonder's own three runs spread 7 points. *** COST UNKNOWN *** Meta publishes no rate for Full - its own model catalog lists the cost as empty and no aggregator carries the model - so the cost column is left blank rather than estimated. Tokens are exact: 108 million across both repos, 104 million of them cache reads. The run used a subscription, so there is no bill either. *** EFFORT max *** the top entry of the effort list Muse Code's model catalog gives this model (minimal to max); not magnitude-probed. The tier describes Muse Code's main loop; its reminder-observer side agent runs at its own fixed efforts. *** RECEIPTS *** Both legs exited 0, run at the same time. Muse Code 1.4.1 (1.4.1-R4503.1). Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 29, 30, 32, 33, 34, 36, 37, 38, 39, 41, 43, 44, 45, 47, 49, 50, 51, 60, 61, 62, 69, 70, 71, 76, 77, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "turns": 346, "turns_repo1": 140, "turns_repo2": 206, "group_rank": 0 }, { "id": "Muse Wonder Full 1.0 (xhigh effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Wonder Full 1.0", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "xhigh", "effort_status": "first_party", "date": "2026-10-13", "color": "#D6336C", "fixed": 44, "repo1_fixed": 16, "repo2_fixed": 28, "partial": 3, "claimed_only": 0, "extras": 35, "false_positive_fixes": 0, "wall_min": 80.3, "cost_usd": null, "cost_kind": "unknown", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-10-14 by the mean of two runs (41.5 of 105). This row reported 44 of 105 from a single run. Nothing here is retracted - it is the receipt for what the board showed before run 2.", "caveat": "One run (44 of 105). Cost unknown: Meta publishes no price for this model. Effort xhigh is one tier below max, not magnitude-probed. Both repos ran at the same time.", "note": "Meta's Muse Wonder Full 1.0 at xhigh effort - one tier below max - through Muse Code, Meta's own agent CLI. ONE RUN: 44 of 105 - 16 of 45 on repo 1 and 28 of 60 on repo 2 - with 3 partial fix(es), 0 claimed-only and 35 genuine unplanted extras; 80.3 minutes. *** AGAINST MAX *** the same model at max, also one run, scored 44 (22 on repo 1, 22 on repo 2) in 138.2 minutes. The totals land together and the split does not: max led on repo 1, xhigh on repo 2. With one run each, the two tiers are not separated here. Muse Wonder 1.0 at max stands at 31.7 (mean of three) and Muse Spark 1.3 at max at 32.2. *** COST UNKNOWN *** Meta publishes no rate for Full - its own model catalog lists the cost as empty - so the cost column is left blank rather than estimated. Tokens are exact: 72 million across both repos. The run used a subscription, so there is no bill either. *** EFFORT xhigh *** a tier in the effort list Muse Code's model catalog gives this model (minimal to max); not magnitude-probed, not the ceiling. The tier describes Muse Code's main loop; its reminder-observer side agent runs at its own fixed efforts. *** RECEIPTS *** Both legs exited 0, run at the same time. Muse Code 1.4.1 (1.4.1-R4503.1). Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 16, 20, 21, 22, 23, 25, 27, 30, 32, 33, 34, 37, 38, 39, 43, 45, 46, 47, 49, 50, 51, 60, 62, 64, 66, 69, 70, 71, 76, 77, 79, 82, 83, 84, 86, 92, 94, 95, 96, 97, 98, 99, 100, 102 ], "turns": 180, "turns_repo1": 65, "turns_repo2": 115, "group_rank": 1 }, { "id": "GPT-5.6 Sol (max effort) - mean of 2", "n_runs": 2, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-16", "color": "#5AB99C", "fixed": 43.5, "repo1_fixed": 19, "repo2_fixed": 24.5, "partial": 2, "claimed_only": 0.5, "extras": 48.5, "false_positive_fixes": 0, "wall_min": 253.1, "cost_usd": 95.35, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "TWO RUNS IS TWO POINTS. The range between them is 7 of 105, which is wider than the gap between several adjacent rows on this board, so treat 43.5 as an estimate with a known spread rather than a settled number - and note the repo figures round independently of the combined mean, so they can sum to 0.1 away from it. The replication is UNVERIFIED: this harness reports no context accounting, so nothing confirms the two runs shared a configuration beyond their identical arm definitions. A third run was attempted and did not finish.", "note": "THE FIRST REPLICATION OF THIS MODEL'S TOP EFFORT SETTING. Runs scored 40 and 47 of 105 (repo 1: 18, 20 - repo 2: 22, 27). Mean 43.5, range 7. *** WHY THIS ROW IS n=2 AND NOT n=3 *** A third run was attempted on 2026-09-16 and did not complete: it ran 156 minutes, finished 563 tool calls, and then the harness account's usage quota was exhausted mid-turn. It produced no report and recorded no tokens and no cost, so there is nothing to score and nothing to average. It is not on this board and is not counted as a low run - a leg that never finished is not a result. Re-running it was declined on cost. *** THE SINGLE-RUN ROW ABOVE SAYS 42 AND THIS MEAN USES 40 FOR THE SAME RUN. BOTH ARE REAL. *** Run 1's transcript has been judged twice. The 2026-07-31 pass scored 19 + 23 = 42 and recorded its judge as grok-4.5 with no served-model field at all, because that column did not exist yet. The 2026-09-13 re-judge scored 18 + 22 = 40 and recorded the model the server actually served, grok-4.6. The judge bridge's model flag is inert - a measurement from 2026-09-04, where grok-4.5, grok-4.6 and a deliberately nonexistent model string all came back as grok-4.6 - so the July pass recorded a REQUEST as though it were a fact and was in all likelihood served the same judge. This mean therefore uses 40: the pass whose judge identity was MEASURED, and the same pass configuration run 2 was scored under. That makes it comparable and documented, NOT more accurate, and the older row keeps its 42 as a receipt rather than being quietly rewritten. *** WHAT THE TWO PASSES BUY, WHICH IS AN ERROR BAR AND NOT SAMPLE SIZE *** They disagree on 3 bugs of 105: on repo 1 one bug moved from a full fix to a partial, on repo 2 two credited bugs became misses. So JUDGE variance on a fixed transcript is 2 points, against MODEL variance of 7 points between the two runs - run-to-run spread is roughly three and a half times the grading noise. Two gradings of one transcript are NOT two runs and were not treated as such; n counts runs of the model. *** THE TWO RUNS COST VERY DIFFERENT AMOUNTS FOR A 7-POINT GAIN *** Run 2 consumed 74 percent more tokens than run 1 (100.9M against 176.0M) and 74 percent more at list rates (69.61 against 121.09 dollars) to score 7 points higher. The runs are six weeks apart on a self-updating CLI that records no version in its transcripts, so whether that is the same model working harder or a changed serving path is NOT established here. Read the mean as a mean of two runs that differed in effort expended as well as in outcome. *** THE REGIME BEHIND THESE TWO RUNS IS UNVERIFIED, NOT VERIFIED *** The automated regime check reads context accounting out of transcripts; this harness reports none - no init record and zero input tokens on every turn - so it returns CANNOT TELL for both repos rather than confirming the runs match. Stated plainly because the alternative is implying a check that did not happen. *** WALL TIME IS NOT COMPARABLE ACROSS THE TWO RUNS *** Run 1's legs ran alone; both of run 2's legs shared the machine with other work and say so in their own rows. Contention can inflate a wall figure and cannot shrink it, so the wall mean here is an upper bound. Scores, tokens and cost are unaffected by it. *** WHAT IS SOLID *** Counts, not inferences. A fixed configuration produced 40 and 47 of 105. Run 2 needed five attempts to produce two clean legs - three earlier legs died on wall budgets or non-zero exits and one was orphaned - so the failure rate of this configuration on this hardware is itself part of the picture and is visible in the metrics receipts.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 25, 26, 27, 28, 29, 30, 33, 34, 35, 38, 39, 40, 41, 43, 44, 45, 46, 47, 49, 50, 53, 55, 59, 60, 62, 64, 65, 70, 71, 76, 77, 78, 80, 82, 83, 84, 91, 92, 93, 95, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "17": 2, "20": 2, "21": 1, "22": 2, "25": 2, "26": 1, "27": 2, "28": 2, "29": 2, "30": 2, "33": 2, "34": 2, "35": 1, "38": 1, "39": 2, "40": 1, "41": 1, "43": 2, "44": 2, "45": 2, "46": 2, "47": 2, "49": 2, "50": 2, "53": 1, "55": 2, "59": 1, "60": 1, "62": 2, "64": 1, "65": 1, "70": 2, "71": 2, "76": 2, "77": 2, "78": 1, "80": 2, "82": 1, "83": 2, "84": 2, "91": 1, "92": 2, "93": 1, "95": 2, "96": 1, "97": 1, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 1082, "turns_repo1": 451, "turns_repo2": 632, "group_rank": 400 }, { "id": "GPT-6 Astra (xhigh effort)", "n_runs": 1, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-04", "color": "#1B9E77", "fixed": 43, "repo1_fixed": 23, "repo2_fixed": 20, "partial": 1, "claimed_only": 1, "extras": 53, "false_positive_fixes": 0, "wall_min": 58.8, "cost_usd": 24.22, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 61, 62, 68, 69, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 159, "turns_repo1": 68, "turns_repo2": 91, "group_rank": 1 }, { "id": "Opus 5.5 (max effort)", "n_runs": 1, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-22", "color": "#DE6C48", "fixed": 43, "repo1_fixed": 18, "repo2_fixed": 25, "partial": 3, "claimed_only": 3, "extras": 9, "false_positive_fixes": 0, "wall_min": 68.5, "cost_usd": 60.49, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (41.0 of 105). This row reported 43 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 39 to 43. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean - and this board has already measured a 13-point spread between two runs of one identical max-effort configuration, so treat 43 as a single draw, not a settled level. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so the gap against them mixes model version with context and is not a clean version comparison. Cost is a list-rate estimate - verified against the harness's own figure to the cent, but not an invoice. The wall figure is contended.", "note": "Anthropic's Opus 5.5 at max effort through Claude Code on the first-party API. 43 of 105: 18 of 45 on repo 1 and 25 of 60 on repo 2, 9 genuine extras, 68.5 minutes, $60.49. ONE RUN. The strongest Anthropic result on this board - the Opus 5 sweep peaked at 27 of 105 (max effort) - and level with the best non-OpenAI rows here. *** READ THE CONTEXT WINDOW BEFORE READING THE GAIN - THIS IS NOT A CLEAN VERSION DELTA *** This arm runs the model string claude-opus-5-5[1m] and therefore a 1M context window: it held a peak of 847,590 tokens on repo 2 and crossed ZERO compaction boundaries, carrying both repos whole through every turn. Every Opus 5 row on this board runs a bare slug, which the CLI treats as a 200K window, and those runs compacted repeatedly mid-task. Two variables moved at once - the model version AND how much of the codebase the model could hold while reasoning about it. The honest claim this row supports is 'Opus 5.5 with a 1M window scores 43', not 'Opus 5.5 is 16 points better than Opus 5'. Separating them needs an Opus 5 arm re-run at [1m], which has not been done. *** THE SUFFIX WAS VERIFIED, NOT ASSUMED *** [1m] is not always honoured: on CLI 2.1.272 the string claude-fable-5-1[1m] was recognised and silently normalised to the bare slug, so an earlier arm on this board requested a window it never received. Probed here before any token was spent, on CLI 2.1.280: init.model returns 'claude-opus-5-5[1m]' verbatim, and the running leg's own init record says the same. *** AUTOCOMPACT RAN AT THE CLI DEFAULT, ALSO DELIBERATE *** A user-level CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=30 had been reaching every leg on this machine, which on a 1M window forces a compaction roughly every 300K tokens - the same setting that turned one earlier arm definition into two different machines six weeks apart. It was removed before this run and its absence re-verified in a leg-shaped probe. *** COST IS A LIST ESTIMATE, BUT A VERIFIED ONE *** $60.49 is token counts at Anthropic's published Opus 5.5 rates: $4 input, $20 output, $0.20 cache read - a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x - and $8 cache write, the 1-hour rate, which is what this harness actually writes. That table reproduces the harness's own computed cost to the cent on both legs (delta $0.00000000), so it is confirmed rather than estimated. It is still not an invoice, hence `list`. The row this arm shipped with initially was 2.02x too high because it inherited Opus 5's price table. The audit that caught it then found the smaller version of the same flaw on fifteen already-published Anthropic rows, which assumed a 5-minute cache write and understated themselves by 5 to 32 per cent; all fifteen were corrected the same day, each one verified against its own stream's ephemeral_1h/ephemeral_5m split rather than by rule - one arm really did write 5-minute caches and kept its lower rate. Long context carries no surcharge - Anthropic prices the full 1M window at standard rates. *** RECEIPTS *** Both legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge returned a malformed empty verdict set on its first attempt and was retried once; the retry is the scored result. The run shared the laptop with benchmark legs from another vendor, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 27, 29, 30, 32, 33, 34, 37, 38, 39, 40, 43, 44, 45, 47, 49, 50, 60, 61, 62, 66, 70, 71, 74, 76, 77, 80, 82, 83, 84, 91, 92, 95, 96, 97, 98, 99, 100, 102 ], "turns": 476, "turns_repo1": 159, "turns_repo2": 317, "group_rank": 200 }, { "id": "Fable 5.1 (max effort)", "n_runs": 1, "model": "Fable 5.1", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-01", "color": "#D9542B", "fixed": 43, "repo1_fixed": 19, "repo2_fixed": 24, "partial": 2, "claimed_only": 4, "extras": 11, "false_positive_fixes": 0, "wall_min": 73.1, "cost_usd": 87.18, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 26, 27, 29, 30, 32, 33, 34, 36, 39, 40, 41, 43, 45, 47, 49, 50, 59, 60, 61, 70, 71, 76, 77, 78, 80, 82, 83, 84, 91, 92, 95, 96, 97, 98, 99, 100, 102 ], "turns": 459, "turns_repo1": 161, "turns_repo2": 298, "group_rank": 0 }, { "id": "GPT-6.1 Sol (xhigh effort) - mean of 3", "n_runs": 3, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-30", "color": "#2BA580", "fixed": 42.7, "repo1_fixed": 22.3, "repo2_fixed": 20.3, "partial": 3.3, "claimed_only": 0, "extras": 56.7, "false_positive_fixes": 0, "wall_min": 98.7, "cost_usd": 4.35, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 41, 43, 44 of 105 - spread 3. Read the mean with that width. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates. Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at xhigh effort through OpenAI's own Codex CLI, on a ChatGPT account, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 41, 43 and 44 of 105 (repo 1: 24, 22, 21 - repo 2: 17, 21, 23). Mean 42.7, range 3. *** THIS SUPERSEDES GPT-6.1 SOL (XHIGH EFFORT) (41 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, pinned CLI, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. *** WALL, COST AND TURNS *** walls 80.6, 98.9, 116.7 minutes; cost $4.68, $4.14, $4.22; turns 211, 189, 201. *** GENUINE EXTRAS *** 63, 46, 61 across the runs, never folded into the score. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at low 31, GPT-6.1 Sol at medium 28, GPT-6.1 Sol at high 38, GPT-6.1 Sol at max 44.3 (mean of 3). At the same tier the board publishes GPT-6 Sol at xhigh 25, GPT-6 Astra at xhigh 43, Sonnet 5.5 at xhigh 36 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at xhigh. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 46, 47, 49, 50, 51, 56, 61, 62, 68, 69, 70, 71, 74, 76, 77, 79, 80, 82, 83, 84, 91, 92, 95, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "17": 2, "20": 3, "21": 3, "22": 3, "23": 2, "25": 3, "26": 3, "27": 3, "28": 3, "29": 3, "30": 3, "32": 1, "33": 3, "34": 3, "35": 1, "36": 1, "37": 3, "39": 3, "40": 3, "41": 3, "43": 3, "44": 3, "45": 3, "46": 1, "47": 3, "49": 3, "50": 3, "51": 3, "56": 1, "61": 1, "62": 3, "68": 1, "69": 3, "70": 1, "71": 2, "74": 1, "76": 3, "77": 3, "79": 1, "80": 2, "82": 2, "83": 3, "84": 3, "91": 1, "92": 3, "95": 2, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 200, "turns_repo1": 95, "turns_repo2": 105, "group_rank": 101 }, { "id": "GPT-5.6 Sol (max effort)", "n_runs": 1, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-08-01", "color": "#5AB99C", "fixed": 42, "repo1_fixed": 19, "repo2_fixed": 23, "partial": 3, "claimed_only": 0, "extras": 40, "false_positive_fixes": 0, "wall_min": 163.8, "cost_usd": 69.61, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-16 by the n=2 MEAN row at the same setting (43.5). This row is one of the two runs behind it and is kept as a receipt. Read its 42 with care: it is the 2026-07-31 judge pass, which recorded its judge as grok-4.5 and had no served-model field at all. The same transcript was re-judged on 2026-09-13 under a configuration that records what the server actually served, and scored 40 - the two passes disagree on 3 bugs of 105. The mean row uses 40 because that is the measured-judge pass and the one the second run was scored under. Nothing here is corrected by arithmetic; the numbers are two gradings of one run, and the mean row's note says which and why.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 25, 27, 28, 29, 30, 33, 34, 35, 39, 41, 43, 44, 45, 46, 47, 49, 50, 55, 59, 60, 62, 70, 71, 76, 77, 78, 80, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": 831, "turns_repo1": 397, "turns_repo2": 434, "group_rank": 400 }, { "id": "Muse Wonder Full 1.0 (max effort, Muse Code, Meta API) - mean of 2", "n_runs": 2, "model": "Muse Wonder Full 1.0", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "first_party", "date": "2026-10-14", "color": "#D6336C", "fixed": 42, "repo1_fixed": 21.5, "repo2_fixed": 20.5, "partial": 2.5, "claimed_only": 0, "extras": 36.5, "false_positive_fixes": 0, "wall_min": 145.3, "cost_usd": null, "cost_kind": "unknown", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Mean of two runs (44 and 40 of 105). Run 2 ran in resumed segments; its time is active time only, and it spans Muse Code 1.4.1 and 1.4.7. Cost unknown: Meta publishes no price for this model.", "note": "Meta's Muse Wonder Full 1.0 at max effort through Muse Code, Meta's own agent CLI. MEAN OF TWO RUNS: 44 and 40 of 105 (repo 1: 22 and 21 of 45 - repo 2: 22 and 19 of 60), mean 42.0. Genuine unplanted extras 53 and 20; claimed-only fixes 0 and 0. Active minutes 138.2 and 152.3. *** XHIGH, ALSO A MEAN OF TWO *** stands at 41.5 (16.0 on repo 1, 25.5 on repo 2). One run each had tied the tiers at 44; two runs each put max narrowly ahead. The tiers split the repos consistently: max leads on repo 1, xhigh on repo 2. *** RUN 2 RAN IN SEGMENTS *** It was interrupted - by the subscription's usage windows, by an operator stop and by a closed console - and each time Muse Code continued the same session. Its wall counts active time only: each segment from its first model call to its last before any refusal; every pause is excluded. Run 2 also straddles a harness update: Muse Code 1.4.1 for its first segment(s), 1.4.7 after the launcher updated itself. *** COST UNKNOWN *** Meta publishes no rate for Full; the cost column is blank, not estimated. *** EFFORT max *** the top of the effort list Muse Code's catalog gives this model; not magnitude-probed. The tier describes Muse Code's main loop. *** RECEIPTS *** 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 36, 37, 38, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 60, 61, 62, 69, 70, 71, 76, 77, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "21": 2, "22": 2, "23": 2, "25": 2, "26": 2, "27": 2, "28": 1, "29": 2, "30": 2, "32": 1, "33": 2, "34": 2, "36": 1, "37": 2, "38": 2, "39": 2, "40": 1, "41": 2, "43": 2, "44": 1, "45": 2, "47": 2, "49": 2, "50": 2, "51": 2, "60": 1, "61": 2, "62": 2, "69": 2, "70": 2, "71": 2, "76": 2, "77": 2, "82": 1, "83": 2, "84": 2, "92": 1, "95": 2, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 356, "turns_repo1": 158, "turns_repo2": 198, "group_rank": 0 }, { "id": "Opus 5.5 (max effort) - mean of 3", "n_runs": 3, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 41.7, "repo1_fixed": 19, "repo2_fixed": 22.7, "partial": 2.4, "claimed_only": 2.6, "extras": 9, "false_positive_fixes": 0, "wall_min": 66.9, "cost_usd": 58.53, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 43, 39 and 43 of 105 - spread 4. Read the mean with that width, not as a point. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at max effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 43, 39 and 43 of 105 (repo 1: 18, 17, 22 of 45 - repo 2: 25, 22, 21 of 60). Mean 41.7, spread 4. *** THIS SUPERSEDES OPUS 5.5 (MAX EFFORT) - MEAN OF 2 (41.0 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 68.5, 60.0, 72.2 minutes, mean 66.9; cost $60.49, $58.39, $56.69, mean $58.52. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 9, 9 and 9 across the runs, mean 9.0 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 32, 33, 34, 36, 37, 38, 39, 40, 41, 43, 44, 45, 47, 49, 50, 60, 61, 62, 66, 70, 71, 74, 76, 77, 80, 82, 83, 84, 91, 92, 95, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 3, "22": 3, "23": 2, "25": 2, "27": 3, "28": 2, "29": 3, "30": 3, "32": 2, "33": 3, "34": 3, "36": 1, "37": 1, "38": 2, "39": 3, "40": 3, "41": 1, "43": 3, "44": 2, "45": 3, "47": 3, "49": 3, "50": 3, "60": 3, "61": 2, "62": 2, "66": 1, "70": 3, "71": 3, "74": 2, "76": 3, "77": 3, "80": 1, "82": 3, "83": 3, "84": 3, "91": 3, "92": 3, "95": 3, "96": 3, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 476, "turns_repo1": 186, "turns_repo2": 289, "group_rank": 200 }, { "id": "Muse Wonder Full 1.0 (xhigh effort, Muse Code, Meta API) - mean of 2", "n_runs": 2, "model": "Muse Wonder Full 1.0", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "xhigh", "effort_status": "first_party", "date": "2026-10-14", "color": "#D6336C", "fixed": 41.5, "repo1_fixed": 16, "repo2_fixed": 25.5, "partial": 2.5, "claimed_only": 0, "extras": 24.5, "false_positive_fixes": 0, "wall_min": 90.5, "cost_usd": null, "cost_kind": "unknown", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Mean of two runs (44 and 39 of 105). Run 2 ran in resumed segments; its time is active time only, and it spans Muse Code 1.4.1 and 1.4.7. Cost unknown: Meta publishes no price for this model.", "note": "Meta's Muse Wonder Full 1.0 at xhigh effort through Muse Code, Meta's own agent CLI. MEAN OF TWO RUNS: 44 and 39 of 105 (repo 1: 16 and 16 of 45 - repo 2: 28 and 23 of 60), mean 41.5. Genuine unplanted extras 35 and 14; claimed-only fixes 0 and 0. Active minutes 80.3 and 100.8. *** MAX, ALSO A MEAN OF TWO *** stands at 42.0 (21.5 on repo 1, 20.5 on repo 2). One run each had tied the tiers at 44; two runs each put max narrowly ahead. The tiers split the repos consistently: max leads on repo 1, xhigh on repo 2. *** RUN 2 RAN IN SEGMENTS *** It was interrupted - by the subscription's usage windows, by an operator stop and by a closed console - and each time Muse Code continued the same session. Its wall counts active time only: each segment from its first model call to its last before any refusal; every pause is excluded. Run 2 also straddles a harness update: Muse Code 1.4.1 for its first segment(s), 1.4.7 after the launcher updated itself. *** COST UNKNOWN *** Meta publishes no rate for Full; the cost column is blank, not estimated. *** EFFORT xhigh *** one tier below the top of the effort list Muse Code's catalog gives this model; not magnitude-probed. The tier describes Muse Code's main loop. *** RECEIPTS *** 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 30, 32, 33, 34, 37, 38, 39, 43, 45, 46, 47, 49, 50, 51, 60, 61, 62, 64, 66, 69, 70, 71, 76, 77, 79, 82, 83, 84, 86, 91, 92, 94, 95, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 2, "20": 2, "21": 2, "22": 2, "23": 1, "25": 2, "26": 1, "27": 2, "30": 2, "32": 1, "33": 2, "34": 2, "37": 2, "38": 2, "39": 2, "43": 2, "45": 2, "46": 1, "47": 2, "49": 2, "50": 2, "51": 2, "60": 1, "61": 1, "62": 2, "64": 1, "66": 2, "69": 2, "70": 2, "71": 2, "76": 2, "77": 2, "79": 1, "82": 2, "83": 2, "84": 2, "86": 1, "91": 1, "92": 2, "94": 1, "95": 2, "96": 1, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 178, "turns_repo1": 72, "turns_repo2": 106, "group_rank": 1 }, { "id": "GPT-6.1 Sol (xhigh effort)", "n_runs": 1, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-29", "color": "#2BA580", "fixed": 41, "repo1_fixed": 24, "repo2_fixed": 17, "partial": 1, "claimed_only": 0, "extras": 63, "false_positive_fixes": 0, "wall_min": 80.6, "cost_usd": 4.68, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 3 (42.7 of 105). This row reported 41 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 41 to 44. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat 41 as one draw. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model; the harness is not held fixed across that comparison. Cost is a list-rate estimate; no request reached the long-context rates (Codex caps the context for this model below 272K). The wall figure may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at xhigh effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 41 of 105: 24 of 45 on repo 1 and 17 of 60 on repo 2, 63 genuine extras, 80.6 minutes, $4.68, 211 turns (91 and 120). ONE RUN, measured straight after the max run, the night the model became reachable. For comparison the board publishes GPT-6.1 Sol at max 44, GPT-6 Sol at xhigh 25, GPT-6 Astra at xhigh 43, Sonnet 5.5 at xhigh 36 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million - half GPT-6 Sol's cached rate. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped either leg - the only load this board's marker records - but the laptop was running several interactive coding-agent sessions that evening, at times near full CPU, so the wall figure may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 56, 62, 68, 69, 76, 77, 83, 84, 92, 98, 99, 100, 102 ], "turns": 211, "turns_repo1": 91, "turns_repo2": 120, "group_rank": 101 }, { "id": "Opus 5.5 (max effort) - mean of 2", "n_runs": 2, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 41, "repo1_fixed": 17.5, "repo2_fixed": 23.5, "partial": 2, "claimed_only": 2, "extras": 9, "false_positive_fixes": 0, "wall_min": 64.3, "cost_usd": 59.45, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (41.7 of 105). This row reported 41.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 39 to 43. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 43 and 39 of 105 - spread 4. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at max effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 43 and 39 of 105 (repo 1: 18, 17 of 45 - repo 2: 25, 22 of 60). Mean 41.0, spread 4. *** THIS SUPERSEDES OPUS 5.5 (MAX EFFORT) (43 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 68.5, 60.0 minutes, mean 64.2; cost $60.49, $58.39, mean $59.44. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 9 and 9 across the runs, mean 9.0 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 32, 33, 34, 37, 38, 39, 40, 43, 44, 45, 47, 49, 50, 60, 61, 62, 66, 70, 71, 74, 76, 77, 80, 82, 83, 84, 91, 92, 95, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "21": 2, "22": 2, "23": 1, "25": 1, "27": 2, "28": 1, "29": 2, "30": 2, "32": 1, "33": 2, "34": 2, "37": 1, "38": 1, "39": 2, "40": 2, "43": 2, "44": 1, "45": 2, "47": 2, "49": 2, "50": 2, "60": 2, "61": 2, "62": 2, "66": 1, "70": 2, "71": 2, "74": 1, "76": 2, "77": 2, "80": 1, "82": 2, "83": 2, "84": 2, "91": 2, "92": 2, "95": 2, "96": 2, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 481, "turns_repo1": 179, "turns_repo2": 302, "group_rank": 200 }, { "id": "Gemini 4 Argon (high effort, Antigravity CLI) - mean of 3", "n_runs": 3, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-10-14", "color": "#2C6FE0", "fixed": 40.7, "repo1_fixed": 13, "repo2_fixed": 27.7, "partial": 1.7, "claimed_only": 1, "extras": 7.3, "false_positive_fixes": 0, "wall_min": 56.5, "cost_usd": 7.16, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Mean of three runs (40, 45 and 37 of 105). Run 2 was cut off by an API billing limit and resumed in the same conversation. The same model through Gemini CLI is a separate row. Cost is a list estimate.", "note": "Google's Gemini 4 Argon at thinking level high - its top tier - through the Antigravity CLI, Google's current agent CLI. MEAN OF THREE RUNS: 40, 45 and 37 of 105 (repo 1: 14, 13 and 12 of 45 - repo 2: 26, 32 and 25 of 60), mean 40.7. Genuine unplanted extras 4, 7 and 11; claimed-only fixes 1, 0 and 2. Walls 37.4, 47.3 and 84.7 minutes; cost $5.74, $6.43 and $9.30 at list, from Google's per-call usage (agy's own token counts read 10-38% lower and were used when this row was first published; corrected 2026-10-14). *** THE SAME MODEL IN GEMINI CLI *** scored 33 in one run at the same thinking level, for $32.08 in 85.1 minutes - kept as its own row. Same model, same tier, same judge. The score gap is all on repo 2, and the Gemini CLI side is a single run while these three spread 8 points (37 to 45), so it is not established as a harness effect. The cost gap is mechanical: Gemini CLI re-sent its whole conversation on every call (44M prompt tokens, 69% cached) and cost about 4.5 times as much. *** HOW ANTIGRAVITY REACHED ARGON *** The Antigravity CLI does not list Argon on a Google AI Pro subscription. Its enterprise gateway mode lists it, signed in with a Gemini API key; a local gateway passes requests through unchanged and presents the key the way the Gemini API expects. The CLI itself sends thinkingLevel HIGH for gemini-4-argon-high, and every request was served by gemini-4-argon. *** EFFORT high IS THE CEILING *** The API accepts low, medium and high for this model and rejects max and ultra. *** COST *** a list estimate from Google's per-call usage at its launch rate ($2 input / $0.10 cached input / $10 output per million tokens, thinking billed as output); 89% of prompt tokens were cache reads. If Google bills prompts over 200K tokens at a higher rate, as it does for its Pro models, this figure and the Gemini CLI row's are low. *** RUN 2 WAS RESUMED *** Both of its legs stopped when the API key's prepaid credit ran out, without a report; after a top-up each continued its own conversation and finished. Its wall is the two segments' sum with the idle gap removed. *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 26, 27, 30, 33, 34, 39, 40, 43, 45, 47, 49, 50, 51, 55, 59, 60, 61, 62, 64, 66, 69, 70, 71, 76, 77, 78, 79, 80, 82, 83, 84, 88, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 105 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 3, "23": 2, "26": 2, "27": 3, "30": 3, "33": 3, "34": 3, "39": 3, "40": 1, "43": 3, "45": 3, "47": 3, "49": 1, "50": 2, "51": 1, "55": 2, "59": 1, "60": 2, "61": 1, "62": 3, "64": 3, "66": 1, "69": 1, "70": 3, "71": 2, "76": 3, "77": 3, "78": 1, "79": 3, "80": 2, "82": 2, "83": 3, "84": 3, "88": 1, "90": 3, "91": 3, "92": 3, "93": 3, "94": 3, "95": 1, "96": 3, "97": 3, "98": 2, "99": 3, "100": 3, "102": 3, "105": 3 }, "coverage_runs": 3, "turns": 470, "turns_repo1": 134, "turns_repo2": 336, "group_rank": 2 }, { "id": "Gemini 4 Argon (medium effort, Antigravity CLI) - mean of 3", "n_runs": 3, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "medium", "effort_status": "verified", "date": "2026-10-14", "color": "#2C6FE0", "fixed": 39.3, "repo1_fixed": 13.7, "repo2_fixed": 25.7, "partial": 1, "claimed_only": 0, "extras": 11.3, "false_positive_fixes": 0, "wall_min": 59.9, "cost_usd": 7.5, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Mean of three runs (33, 39 and 46 of 105). Medium is the model's default tier. Cost is a list estimate from Google's per-call usage.", "note": "Google's Gemini 4 Argon at thinking level medium - its default, one tier below the ceiling - through the Antigravity CLI, set up exactly as the high row. MEAN OF THREE RUNS: 33, 39 and 46 of 105 (repo 1: 13, 15 and 13 of 45 - repo 2: 20, 24 and 33 of 60), mean 39.3. Genuine unplanted extras 8, 13 and 13; claimed-only fixes 0, 0 and 0. Walls 36.3, 44.7 and 98.5 minutes; cost $6.81, $6.69 and $9.01. *** AGAINST HIGH *** the high row stands at 40.7 (mean of three). The two tiers are level on repo 1; high leads on repo 2 on average, and medium costs about the same per run. The runs spread widely (33 to 46), so the gap between the tiers is smaller than that spread. *** EFFORT medium *** agy sends thinkingLevel MEDIUM for gemini-4-argon-medium and the gateway read it back on the main loop's calls; a couple of agy's own side calls per run go out at HIGH regardless of the setting. *** RUN 3, REPO 1 WAS RESUMED *** a console Ctrl+C killed its local gateway 46 minutes in; the same conversation continued once the gateway was back, for 12.7 more minutes. Its wall is the two segments' sum. *** COST *** a list estimate from Google's per-call usage at its launch rate ($2 input / $0.10 cached input / $10 output per million tokens); legs that ran together are split by agy's per-leg share. If Google bills prompts over 200K tokens at a higher rate, as it does for its Pro models, this figure is low. *** RECEIPTS *** 5 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 26, 27, 30, 33, 34, 37, 39, 40, 43, 45, 47, 49, 50, 51, 53, 55, 60, 61, 62, 64, 66, 69, 70, 71, 76, 77, 78, 79, 80, 82, 83, 84, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 105 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 3, "23": 3, "26": 3, "27": 3, "30": 3, "33": 3, "34": 2, "37": 1, "39": 3, "40": 1, "43": 3, "45": 3, "47": 3, "49": 2, "50": 3, "51": 1, "53": 1, "55": 1, "60": 3, "61": 1, "62": 3, "64": 3, "66": 1, "69": 1, "70": 3, "71": 1, "76": 3, "77": 3, "78": 1, "79": 2, "80": 3, "82": 3, "83": 3, "84": 3, "91": 2, "92": 3, "93": 3, "94": 2, "95": 1, "96": 2, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3, "105": 1 }, "coverage_runs": 3, "turns": 437, "turns_repo1": 138, "turns_repo2": 299, "group_rank": 3 }, { "id": "GPT-5.6 Sol (xhigh effort)", "n_runs": 1, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-08-28", "color": "#5AB99C", "fixed": 39, "repo1_fixed": 18, "repo2_fixed": 21, "partial": 3, "claimed_only": 0, "extras": 59, "false_positive_fixes": 0, "wall_min": 126.2, "cost_usd": 52.75, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": "Second only to Sol max (42) on the board; the Sol dial runs high 34 -> xhigh 39 -> max 42.", "fixed_bugs": [ 13, 16, 17, 20, 22, 25, 27, 28, 29, 30, 33, 34, 35, 37, 39, 43, 44, 45, 46, 47, 49, 50, 53, 55, 60, 62, 64, 71, 76, 77, 80, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 485, "turns_repo1": 233, "turns_repo2": 252, "group_rank": 401 }, { "id": "Sonnet 5.5 (xhigh effort)", "n_runs": 1, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 39, "repo1_fixed": 18, "repo2_fixed": 21, "partial": 2, "claimed_only": 1, "extras": 19, "false_positive_fixes": 0, "wall_min": 122.4, "cost_usd": 60.3, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-29 by the mean of 3 (36.0 of 105). This row reported 39 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 34 to 39. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat this as one draw. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at xhigh effort through Claude Code on the first-party API, measured the evening it shipped. 39 of 105: 18 of 45 on repo 1 and 21 of 60 on repo 2, 19 genuine extras, 122.4 minutes, $60.30. ONE RUN. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** IT KEEPS WORKING AFTER IT WRITES ITS REPORT *** On repo 1 it took 307 turns (the harness's own count); it first wrote its report 39 minutes in and its last model call came at minute 95 - 56 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 281 turns (the harness's own count); it first wrote its report 27 minutes in and its last model call came at minute 28. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. Its own repo-1 report says each fix carries a regression test that it confirmed fails with the fix reverted - a mutation check - which is where the extra turns went. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 935,562 tokens; 0 compactions across 2 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge returned an empty verdict set 2 times and was re-run - the re-run is the scored result. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 33, 34, 39, 40, 43, 44, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 79, 80, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100 ], "turns": 588, "turns_repo1": 307, "turns_repo2": 281, "group_rank": 501 }, { "id": "GPT-6.1 Sol (high effort)", "n_runs": 1, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-09-29", "color": "#2BA580", "fixed": 38, "repo1_fixed": 15, "repo2_fixed": 23, "partial": 2, "claimed_only": 0, "extras": 45, "false_positive_fixes": 0, "wall_min": 78.7, "cost_usd": 3.51, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 2 (36.5 of 105). This row reported 38 of 105 from a single run. Its note announced two replicates; this tier was then cut to one, so the mean is of 2. The replicate is a byte-identical re-run; across 2 the runs span 35 to 38. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat 38 as one draw until the replicates land. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates (Codex caps the context for this model below 272K). Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at high effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 38 of 105: 15 of 45 on repo 1 and 23 of 60 on repo 2, 45 genuine extras, 78.7 minutes, $3.51, 159 turns (53 and 106). ONE RUN - two replicates follow and will supersede this row. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at xhigh 41, GPT-6.1 Sol at max 44. At the same tier the board publishes GPT-6 Sol at high 20, GPT-6 Astra at high 35, Sonnet 5.5 at high 34.3 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at high. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 25, 27, 29, 33, 34, 35, 37, 39, 43, 44, 45, 46, 47, 49, 50, 51, 56, 59, 62, 67, 69, 76, 77, 78, 80, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": 159, "turns_repo1": 53, "turns_repo2": 106, "group_rank": 102 }, { "id": "GPT-6.1 Sol (high effort) - mean of 2", "n_runs": 2, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-09-30", "color": "#2BA580", "fixed": 36.5, "repo1_fixed": 17, "repo2_fixed": 19.5, "partial": 2.5, "claimed_only": 0.5, "extras": 40, "false_positive_fixes": 0, "wall_min": 70.3, "cost_usd": 3.26, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Two runs of an identical configuration scoring 38 and 35 of 105 - spread 3. Read the mean with that width - two runs is a thin mean. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates. Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at high effort through OpenAI's own Codex CLI, on a ChatGPT account, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 38 and 35 of 105 (repo 1: 15, 19 - repo 2: 23, 16). Mean 36.5, range 3. *** THIS SUPERSEDES GPT-6.1 SOL (HIGH EFFORT) (38 OF 105) *** *** THE REPLICATE IS BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, pinned CLI, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. *** WALL, COST AND TURNS *** walls 78.7, 61.9 minutes; cost $3.51, $3.02; turns 159, 131. *** GENUINE EXTRAS *** 45, 35 across the runs, never folded into the score. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at low 31, GPT-6.1 Sol at medium 28, GPT-6.1 Sol at xhigh 42.7 (mean of 3), GPT-6.1 Sol at max 44.3 (mean of 3). At the same tier the board publishes GPT-6 Sol at high 20, GPT-6 Astra at high 35, Sonnet 5.5 at high 34.3 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at high. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 26, 27, 29, 30, 33, 34, 35, 37, 39, 40, 41, 43, 44, 45, 46, 47, 49, 50, 51, 56, 59, 62, 66, 67, 69, 76, 77, 78, 80, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "21": 1, "22": 2, "25": 2, "26": 1, "27": 2, "29": 2, "30": 1, "33": 2, "34": 2, "35": 1, "37": 2, "39": 2, "40": 1, "41": 1, "43": 2, "44": 2, "45": 2, "46": 1, "47": 2, "49": 2, "50": 2, "51": 1, "56": 1, "59": 1, "62": 2, "66": 1, "67": 1, "69": 2, "76": 2, "77": 2, "78": 1, "80": 1, "82": 2, "83": 2, "84": 2, "92": 2, "95": 1, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 145, "turns_repo1": 54, "turns_repo2": 92, "group_rank": 102 }, { "id": "Gemini 4 Argon (medium effort, Antigravity CLI) - mean of 2", "n_runs": 2, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "medium", "effort_status": "verified", "date": "2026-10-14", "color": "#2C6FE0", "fixed": 36, "repo1_fixed": 14, "repo2_fixed": 22, "partial": 0.5, "claimed_only": 0, "extras": 10.5, "false_positive_fixes": 0, "wall_min": 40.5, "cost_usd": 6.75, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-10-14 by the mean of three runs (39.3 of 105). This row reported 36.0 of 105 as the mean of 2 runs. Nothing here is retracted - it is the receipt for what the board showed before run 3.", "caveat": "Mean of two runs (33 and 39 of 105). Medium is the model's default tier. Cost is a list estimate from Google's per-call usage.", "note": "Google's Gemini 4 Argon at thinking level medium - its default, one tier below the ceiling - through the Antigravity CLI, set up exactly as the high row. MEAN OF TWO RUNS: 33 and 39 of 105 (repo 1: 13 and 15 of 45 - repo 2: 20 and 24 of 60), mean 36.0. Genuine unplanted extras 8 and 13; claimed-only fixes 0 and 0. Walls 36.3 and 44.7 minutes; cost $6.81 and $6.69. *** AGAINST HIGH *** the high row stands at 40.7 (mean of three). The two tiers are level on repo 1 and the gap is on repo 2; medium costs about the same as high per run, so the higher tier is the better buy here. *** EFFORT medium *** agy sends thinkingLevel MEDIUM for gemini-4-argon-medium and the gateway read it back on the main loop's calls; a couple of agy's own side calls per run go out at HIGH regardless of the setting. *** COST *** a list estimate from Google's per-call usage at its launch rate ($2 input / $0.10 cached input / $10 output per million tokens); legs that ran together are split by agy's per-leg share. If Google bills prompts over 200K tokens at a higher rate, as it does for its Pro models, this figure is low. *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 26, 27, 30, 33, 34, 39, 40, 43, 45, 47, 49, 50, 60, 62, 64, 70, 76, 77, 79, 80, 82, 83, 84, 91, 92, 93, 94, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "21": 1, "22": 2, "23": 2, "26": 2, "27": 2, "30": 2, "33": 2, "34": 2, "39": 2, "40": 1, "43": 2, "45": 2, "47": 2, "49": 2, "50": 2, "60": 2, "62": 2, "64": 2, "70": 2, "76": 2, "77": 2, "79": 1, "80": 2, "82": 2, "83": 2, "84": 2, "91": 1, "92": 2, "93": 2, "94": 1, "96": 1, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 450, "turns_repo1": 123, "turns_repo2": 327, "group_rank": 3 }, { "id": "Opus 5.5 (xhigh effort) - mean of 3", "n_runs": 3, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 36, "repo1_fixed": 17, "repo2_fixed": 19, "partial": 3.3, "claimed_only": 1, "extras": 13.3, "false_positive_fixes": 0, "wall_min": 42.5, "cost_usd": 34.98, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 35, 37 and 36 of 105 - spread 2. Read the mean with that width, not as a point. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at xhigh effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 35, 37 and 36 of 105 (repo 1: 16, 19, 16 of 45 - repo 2: 19, 18, 20 of 60). Mean 36.0, spread 2. *** THIS SUPERSEDES OPUS 5.5 (XHIGH EFFORT) - MEAN OF 2 (36.0 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 43.7, 45.6, 38.4 minutes, mean 42.6; cost $35.14, $36.74, $33.06, mean $34.98. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 14, 17 and 9 across the runs, mean 13.3 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 32, 33, 34, 36, 39, 40, 41, 43, 44, 45, 46, 47, 49, 50, 60, 62, 70, 71, 76, 77, 80, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 2, "22": 3, "23": 1, "25": 3, "27": 2, "28": 3, "29": 3, "30": 3, "32": 1, "33": 3, "34": 3, "36": 1, "39": 3, "40": 2, "41": 2, "43": 3, "44": 1, "45": 3, "46": 1, "47": 3, "49": 3, "50": 3, "60": 3, "62": 3, "70": 1, "71": 3, "76": 3, "77": 3, "80": 2, "82": 2, "83": 3, "84": 3, "91": 1, "92": 3, "95": 2, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 362, "turns_repo1": 134, "turns_repo2": 228, "group_rank": 201 }, { "id": "Opus 5.5 (xhigh effort) - mean of 2", "n_runs": 2, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 36, "repo1_fixed": 17.5, "repo2_fixed": 18.5, "partial": 3.5, "claimed_only": 1, "extras": 15.5, "false_positive_fixes": 0, "wall_min": 44.6, "cost_usd": 35.94, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (36.0 of 105). This row reported 36.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 35 to 37. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 35 and 37 of 105 - spread 2. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at xhigh effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 35 and 37 of 105 (repo 1: 16, 19 of 45 - repo 2: 19, 18 of 60). Mean 36.0, spread 2. *** THIS SUPERSEDES OPUS 5.5 (XHIGH EFFORT) (35 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 43.7, 45.6 minutes, mean 44.7; cost $35.14, $36.74, mean $35.94. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 14 and 17 across the runs, mean 15.5 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 27, 28, 29, 30, 32, 33, 34, 36, 39, 40, 41, 43, 44, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 80, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "21": 1, "22": 2, "25": 2, "27": 2, "28": 2, "29": 2, "30": 2, "32": 1, "33": 2, "34": 2, "36": 1, "39": 2, "40": 2, "41": 1, "43": 2, "44": 1, "45": 2, "47": 2, "49": 2, "50": 2, "60": 2, "62": 2, "70": 1, "71": 2, "76": 2, "77": 2, "80": 1, "82": 1, "83": 2, "84": 2, "92": 2, "95": 2, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 362, "turns_repo1": 130, "turns_repo2": 232, "group_rank": 201 }, { "id": "Sonnet 5.5 (xhigh effort) - mean of 3", "n_runs": 3, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-29", "color": "#E68F74", "fixed": 36, "repo1_fixed": 16, "repo2_fixed": 20, "partial": 3.3, "claimed_only": 3.3, "extras": 14, "false_positive_fixes": 0, "wall_min": 72.7, "cost_usd": 42.86, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 39, 34 and 35 of 105 - spread 5. Read the mean with that width. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at xhigh effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 39, 34 and 35 of 105 (repo 1: 18, 16, 14 - repo 2: 21, 18, 21). Mean 36.0, range 5. *** THIS SUPERSEDES SONNET 5.5 (XHIGH EFFORT) (39 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at xhigh effort 36 (mean of 3) of 105. *** WALL AND COST *** walls 122.4, 44.7, 51.0 minutes, mean 72.7; cost $60.30, $36.30, $31.97, mean $42.86. *** GENUINE EXTRAS *** 19, 12 and 11 across the runs, mean 14.0, never folded into the score. *** IT KEEPS WORKING AFTER IT WRITES ITS REPORT *** On repo 1 it took 307, 201, 141 turns (the harness's own count); it first wrote its report 39, 25, 22 minutes in and its last model call came at minute 95, 26, 22 - 56, 0, 0 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at xhigh took 122-142 turns there over three runs of 15-21 minutes, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 281, 228, 347 turns (the harness's own count); it first wrote its report 27, 19, 28 minutes in and its last model call came at minute 28, 19, 29 - 1, 0, 1 minutes of further checking and revising after the report existed. For scale: Opus 5.5 at xhigh took 220-232 turns there over three runs of 22-26 minutes, and every one of those runs wrote its report within 1 minute of its last call. Its own repo-1 report says each fix carries a regression test that it confirmed fails with the fix reverted - a mutation check - which is where the extra turns went. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 996,172 tokens; 0 compactions across 6 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The repo 2 judge on run 1 returned an empty verdict set 2 times and was re-run - the re-run is the scored result. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 33, 34, 36, 39, 40, 43, 44, 45, 47, 49, 50, 59, 60, 61, 62, 70, 71, 76, 77, 78, 79, 80, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 2, "22": 3, "23": 2, "25": 3, "27": 3, "28": 3, "29": 3, "30": 3, "33": 2, "34": 2, "36": 1, "39": 3, "40": 1, "43": 3, "44": 2, "45": 3, "47": 3, "49": 3, "50": 3, "59": 1, "60": 3, "61": 2, "62": 2, "70": 3, "71": 3, "76": 3, "77": 3, "78": 1, "79": 1, "80": 2, "82": 2, "83": 3, "84": 3, "91": 2, "92": 2, "95": 2, "97": 3, "98": 3, "99": 3, "100": 3, "102": 1 }, "coverage_runs": 3, "turns": 502, "turns_repo1": 216, "turns_repo2": 285, "group_rank": 501 }, { "id": "Gemini 4 Argon (low effort, Antigravity CLI) - mean of 3", "n_runs": 3, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "low", "effort_status": "verified", "date": "2026-10-14", "color": "#2C6FE0", "fixed": 35.3, "repo1_fixed": 12.7, "repo2_fixed": 22.7, "partial": 0.7, "claimed_only": 0, "extras": 5.7, "false_positive_fixes": 0, "wall_min": 59.6, "cost_usd": 6.47, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Mean of three runs (36, 32 and 38 of 105). Judged by Grok, not Codex like the other Argon rows. Two legs ran in resumed segments; their time is active time only. Cost is a list estimate from Google's per-call usage.", "note": "Google's Gemini 4 Argon at thinking level low - the lowest tier the API accepts for it - through the Antigravity CLI, set up exactly as the medium and high rows. MEAN OF THREE RUNS: 36, 32 and 38 of 105 (repo 1: 12, 12 and 14 of 45 - repo 2: 24, 20 and 24 of 60), mean 35.3. Genuine unplanted extras 6, 3 and 8; claimed-only fixes 0, 0 and 0. Active minutes 78.2, 44.9 and 55.7; cost $6.24, $5.97 and $7.19. Medium stands at 39.3 and high at 40.7 (means of three): the tiers order as expected, the dial acts mostly on repo 2, and low is only a little cheaper. *** TWO LEGS RAN IN SEGMENTS *** In runs 1 and 3 the repo-2 leg was interrupted - by a console Ctrl+C, by the local gateway holding headers until Google started streaming (the CLI gives up after about a minute), and by the API key's depleted prepaid credit - and each time continued the same conversation. Their time counts active work only: each segment from its first successful model call to its last. *** A DIFFERENT JUDGE *** all six legs were judged by Grok (grok-4.7), not Codex gpt-5.5 like the other Argon rows: from about 16:56 on Oct 14 Codex refused gpt-5.5 for the account. Grok is this board's second calibrated judge and not a sibling of Gemini, but scores from two judges are not perfectly interchangeable. *** EFFORT low *** agy sends thinkingLevel LOW for gemini-4-argon-low, read back at the gateway; a couple of agy's own side calls per run go out at HIGH regardless. *** COST *** a list estimate from Google's per-call usage at its launch rate ($2 input / $0.10 cached input / $10 output per million tokens), split between legs by agy's per-leg share. *** RECEIPTS *** 3 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 26, 27, 30, 33, 34, 39, 43, 45, 47, 49, 50, 51, 59, 60, 61, 62, 64, 69, 70, 71, 76, 77, 78, 79, 82, 83, 84, 87, 91, 92, 93, 94, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 3, "23": 3, "26": 3, "27": 1, "30": 3, "33": 3, "34": 3, "39": 3, "43": 3, "45": 3, "47": 3, "49": 1, "50": 3, "51": 1, "59": 2, "60": 3, "61": 1, "62": 3, "64": 1, "69": 1, "70": 3, "71": 1, "76": 3, "77": 3, "78": 2, "79": 1, "82": 3, "83": 3, "84": 3, "87": 1, "91": 1, "92": 3, "93": 3, "94": 3, "96": 1, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 483, "turns_repo1": 134, "turns_repo2": 349, "group_rank": 4 }, { "id": "GPT-6 Astra (high effort)", "n_runs": 1, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-09-04", "color": "#1B9E77", "fixed": 35, "repo1_fixed": 19, "repo2_fixed": 16, "partial": 1, "claimed_only": 2, "extras": 40, "false_positive_fixes": 0, "wall_min": 40.5, "cost_usd": 20.6, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 68, 71, 76, 77, 80, 82, 83, 84, 92, 98, 99, 100, 102 ], "turns": 131, "turns_repo1": 73, "turns_repo2": 58, "group_rank": 2 }, { "id": "Opus 5.5 (xhigh effort)", "n_runs": 1, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-22", "color": "#DE6C48", "fixed": 35, "repo1_fixed": 16, "repo2_fixed": 19, "partial": 3, "claimed_only": 2, "extras": 14, "false_positive_fixes": 0, "wall_min": 43.7, "cost_usd": 35.14, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (36.0 of 105). This row reported 35 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 35 to 37. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean. This is the xhigh rung of a four-rung same-day sweep (medium 30, high 32, xhigh 35, max 43 of 105); the ladder is monotone, but no gap in it exceeds the 13-point run-to-run spread this board has measured on a single identical configuration, so read the direction, not the ordering of adjacent rungs. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. The wall figure is contended.", "note": "Anthropic's Opus 5.5 at xhigh effort through Claude Code on the first-party API. 35 of 105: 16 of 45 on repo 1 and 19 of 60 on repo 2, 14 genuine extras, 43.7 minutes, $35.14. ONE RUN. *** THIS IS ONE RUNG OF A FOUR-RUNG SWEEP *** All four Opus 5.5 rows on this board are the same arm definition with one character changed - the effort dial - run on the same day, on the same machine, against the same two repos, judged the same way. The ladder is monotone: medium 30, high 32, xhigh 35, max 43 of 105, costing $16.55, $23.03, $35.14 and $60.49. *** BUT READ THE LADDER AS A TREND, NOT AS FOUR LEVELS *** every rung is a single run, and this board has already measured a 13-POINT spread between two runs of one identical max-effort configuration elsewhere. No gap on this ladder is that wide - the largest, max over xhigh, is 8. So the direction is credible across four points; the ordering of any two adjacent rungs is not resolved at n=1, and nothing here says xhigh is reliably worse than the rung above it. *** THE DIAL BINDS - MEASURED, NOT ASSUMED *** this board has caught several third-party routes that accept a reasoning field and ignore it, so 'first party' is not taken on trust. Token spend on repo 2 rises monotonically with the dial across the sweep - 33.4M at medium, 46.8M at high, 82.7M at xhigh, 150.2M at max, a 4.5x span - and wall rises with it. This rung burned 82.7M. Whatever else is true, the setting is reaching the model and changing how hard it works. *** THE CONTEXT WINDOW, WHICH IS WHY THESE ROWS DO NOT COMPARE TO THE OPUS 5 LADDER *** this arm runs the model string claude-opus-5-5[1m] and therefore a 1M window: it held a peak of 626,627 tokens on repo 2 and crossed ZERO compaction boundaries, carrying the repo whole through every turn. Every Opus 5 row on this board runs a bare slug, which the CLI treats as 200K, and those runs compacted repeatedly mid-task. Comparing this ladder to the Opus 5 ladder moves two variables at once - model version AND how much of the codebase the model could hold while reasoning about it. The [1m] suffix was verified on CLI 2.1.280 rather than assumed (an earlier arm on this board requested a 1M window and was silently normalised to the bare slug on 2.1.272); init.model returns 'claude-opus-5-5[1m]' verbatim. Autocompact ran at the CLI's own default - a user-level CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=30 that had been reaching legs on this machine was removed before the sweep and its absence re-verified in a leg-shaped probe. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Opus 5.5 rates: $4 input, $20 output, $0.20 cache read - a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x - and $8 cache write, the 1-hour rate, which is what this harness actually writes. Not an invoice, hence `list`. *** RECEIPTS *** Both legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The run shared the laptop with benchmark legs from another vendor, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 25, 27, 28, 29, 30, 33, 34, 39, 40, 41, 43, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 80, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "turns": 353, "turns_repo1": 122, "turns_repo2": 231, "group_rank": 201 }, { "id": "Sonnet 5.5 (high effort) - mean of 3", "n_runs": 3, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-29", "color": "#E68F74", "fixed": 34.3, "repo1_fixed": 15.7, "repo2_fixed": 18.7, "partial": 1, "claimed_only": 2, "extras": 10.7, "false_positive_fixes": 0, "wall_min": 27.9, "cost_usd": 19.85, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 32, 34 and 37 of 105 - spread 5. Read the mean with that width. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at high effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 32, 34 and 37 of 105 (repo 1: 15, 16, 16 - repo 2: 17, 18, 21). Mean 34.3, range 5. *** THIS SUPERSEDES SONNET 5.5 (HIGH EFFORT) (32 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at high effort 31.7 (mean of 3) of 105. *** WALL AND COST *** walls 24.5, 25.5, 33.9 minutes, mean 28.0; cost $16.73, $17.16, $25.67, mean $19.85. *** GENUINE EXTRAS *** 12, 8 and 12 across the runs, mean 10.7, never folded into the score. *** HOW IT SPENT THE RUN *** On repo 1 it took 138, 99, 111 turns (the harness's own count); it first wrote its report 16, 13, 11 minutes in and its last model call came at minute 16, 13, 11. For scale: Opus 5.5 at high took 81-107 turns there over three runs of 8-11 minutes, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 156, 118, 200 turns (the harness's own count); it first wrote its report 8, 12, 22 minutes in and its last model call came at minute 8, 12, 22. For scale: Opus 5.5 at high took 162-190 turns there over three runs of 13-16 minutes, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 803,795 tokens; 0 compactions across 6 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 39, 40, 43, 45, 47, 49, 50, 60, 61, 70, 71, 76, 77, 80, 82, 83, 84, 91, 92, 95, 96, 97, 98, 99, 100 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 2, "22": 2, "23": 1, "25": 3, "26": 2, "27": 3, "28": 3, "29": 3, "30": 3, "33": 3, "34": 3, "39": 3, "40": 1, "43": 3, "45": 3, "47": 3, "49": 3, "50": 3, "60": 3, "61": 3, "70": 3, "71": 3, "76": 3, "77": 3, "80": 1, "82": 3, "83": 3, "84": 3, "91": 2, "92": 3, "95": 1, "96": 1, "97": 3, "98": 3, "99": 3, "100": 3 }, "coverage_runs": 3, "turns": 274, "turns_repo1": 116, "turns_repo2": 158, "group_rank": 502 }, { "id": "GPT-6 Astra (medium effort)", "n_runs": 1, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-05", "color": "#1B9E77", "fixed": 34, "repo1_fixed": 19, "repo2_fixed": 15, "partial": 1, "claimed_only": 1, "extras": 33, "false_positive_fixes": 0, "wall_min": 28.0, "cost_usd": 15.78, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 17, 20, 22, 25, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 62, 73, 74, 76, 77, 82, 83, 84, 97, 98, 99, 100 ], "turns": 104, "turns_repo1": 54, "turns_repo2": 50, "group_rank": 3 }, { "id": "GPT-5.6 Sol (re-run Jul 31)", "n_runs": 1, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-07-31", "color": "#5AB99C", "fixed": 34, "repo1_fixed": 13, "repo2_fixed": 21, "partial": 4, "claimed_only": 0, "extras": 28, "false_positive_fixes": 0, "wall_min": 66.7, "cost_usd": 33.92, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 29, 30, 33, 34, 39, 44, 45, 46, 47, 49, 50, 62, 66, 70, 76, 77, 80, 81, 82, 83, 84, 92, 95, 96, 97, 98, 99, 102 ], "turns": 269, "turns_repo1": 129, "turns_repo2": 140, "group_rank": 402 }, { "id": "Muse Spark 1.3 (max effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-14", "color": "#E57EA2", "fixed": 33.7, "repo1_fixed": 15.3, "repo2_fixed": 18.3, "partial": 2, "claimed_only": 1.3, "extras": 26.3, "false_positive_fixes": 0, "wall_min": 89.8, "cost_usd": 19.16, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-17 by the n=5 MEAN row at the same setting (32.2). Two further runs of the identical configuration scored 29 and 31, so this row's three runs were the top of the distribution rather than the middle of it. Its 33.7 is not wrong and has not been rewritten - it is what three runs of this arm said. What changed is the error bar: range 2 across three runs, range 6 across five. Kept as a receipt precisely because the difference between the two rows is the point.", "caveat": "Every figure on this row is a MEAN of three runs, carried to one decimal - 33.7 of 105, not a 34 that happens to be an average. The three runs span 33 to 35. This row SUPERSEDES the single-run max row (33/105) published 2026-09-13, which sat 0.7 below the mean; the supersession is about method, not a correction. The two repo figures are rounded independently and sum to 33.6 rather than 33.7 - the combined figure is the mean of the three TOTALS, which is the one that reconciles with the runs; a tenth of a bug is the cost of a fixed precision and is not hidden. THE THREE RUNS DIFFER IN HARNESS VERSION (1.1.1 for the first, 1.2.1 for the other two), which no other replicated rung on this board does - see the note. With this row Meta's dial is the first on this board measured at n=3 on every rung.", "note": "THE TOP RUNG OF META'S DIAL, RUN THREE TIMES - and with it, the FIRST EFFORT DIAL ON THIS BOARD REPLICATED END TO END. Runs scored 33, 33 and 35 of 105 (repo 1: 14, 17, 15 - repo 2: 19, 16, 20). Mean 33.7, range 2. *** THE WHOLE DIAL, ALL FIVE RUNGS AT n=3 *** low 9.7, medium 13.0, high 18.7, xhigh 20.3, max 33.7. Monotone, with no dip and no dead middle. Three-run ranges, in the same order: 9, 6, 3, 9, 2. *** WHAT THAT DOES AND DOES NOT LICENSE *** Two adjacent rungs are distinguishable here only where the step clears the run-to-run range at both ends. max over xhigh is a 13.4-point step against ranges of 9 and 2, and is the one gap on this dial that no amount of run-to-run noise explains away. high over medium (5.7 against ranges of 6 and 3) and xhigh over high (under two points against 3 and 9) are NOT separated by anything this board can measure. What is established is the ORDER of the endpoints and the size of the top step; what is not established is that every rung is a rung. *** THE THREE RUNS ARE NOT HARNESS-IDENTICAL, AND THAT IS A REAL ASTERISK *** Run 1 ran Muse Code 1.1.1 (1.1.1-R2514.1); runs 2 and 3 ran 1.2.1 (1.2.1-R2847.1). Muse Code self-updates and did so between waves. Every other replicated rung on this board held the harness version fixed across all three runs, so this rung's mean carries one source of variation the others do not. It is not a reason to discount the row - 1.1.1 scored 33 and the 1.2.1 pair scored 33 and 35, so the update did not move the number in any direction this sample can see - but a mean is a claim that its members differ only by chance, and here they also differ by a version string. Stated so nobody has to reconstruct it from two CSVs. *** WHAT IS SOLID *** Counts, not inferences. A fixed configuration produced 33, 33 and 35 on this rung - the TIGHTEST spread of any rung on this dial, a range of 2 against 9 at the bottom. Across the three runs this rung ever fixed 45 distinct bugs of 105, well above the 33.7 any single run manages: 25 of those 45 fell in all three runs, 6 in two of them and 14 in exactly one. The gap between 'ever' and 'reliably' is the thing a single run cannot show. All six legs were clean: the Linux install returned 0, the shim logged zero errors, every leg exited 0, and no leg carries a handicap note. Per-leg receipts (repo 1 / repo 2): shim calls 471/557, 467/351, 363/484; wall 102.3, 86.6 and 80.4 minutes; cost 19.83, 19.83 and 17.81 dollars. Genuine extras 16, 25 and 38 - a range of 22 on a count that is not scored and is not meant to be, and a useful reminder of how noisy the unplanted column is at n=1. Claimed-only is 0 on five of the six legs; run 2's repo-2 leg claimed 4 fixes the judge did not confirm. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR: Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control. billing=LIST ESTIMATE, NOT A BILL: a subscription account with no per-run credits delta, costed at the same Meta list rates as the other first-party rows, which is what keeps them comparable. ctx 1M.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 56, 60, 62, 69, 70, 71, 76, 77, 80, 82, 83, 84, 87, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 3, "21": 3, "22": 3, "23": 3, "25": 2, "26": 1, "27": 2, "28": 1, "29": 2, "30": 3, "33": 3, "34": 3, "39": 3, "40": 1, "41": 1, "43": 3, "44": 2, "45": 3, "47": 3, "49": 3, "50": 3, "51": 1, "56": 1, "60": 3, "62": 3, "69": 1, "70": 1, "71": 3, "76": 3, "77": 3, "80": 1, "82": 1, "83": 3, "84": 3, "87": 1, "91": 1, "92": 3, "95": 1, "97": 3, "98": 3, "99": 3, "100": 1, "102": 3 }, "coverage_runs": 3, "turns": 455, "turns_repo1": 176, "turns_repo2": 279, "group_rank": 200 }, { "id": "GPT-5.6 Luna (max effort)", "n_runs": 1, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-07-31", "color": "#79C6AF", "fixed": 33, "repo1_fixed": 17, "repo2_fixed": 16, "partial": 4, "claimed_only": 0, "extras": 31, "false_positive_fixes": 0, "wall_min": 85.7, "cost_usd": 1.8, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-22 by the n=2 row at the same configuration (31.5 of 105). This row was a SINGLE run at 33, and it was featured, so it was the first number most readers met. Two runs of the same configuration scored 33 and 30. Nothing it claimed is retracted; it said n=1 and it was one run. Kept rather than deleted because it is the receipt for what the board showed before this replicate.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 27, 29, 30, 33, 34, 37, 39, 40, 43, 44, 45, 46, 47, 49, 50, 62, 76, 77, 80, 83, 84, 91, 92, 97, 98, 99, 100 ], "turns": 616, "turns_repo1": 93, "turns_repo2": 523, "group_rank": 600 }, { "id": "Muse Spark 1.3 (max effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-13", "color": "#E57EA2", "fixed": 33, "repo1_fixed": 14, "repo2_fixed": 19, "partial": 3, "claimed_only": 0, "extras": 16, "false_positive_fixes": 0, "wall_min": 102.3, "cost_usd": 19.83, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same rung (33.7). This row is one of three runs that scored 33, 33 and 35; it is kept because a single run is no longer how this rung is reported, and nothing in it is corrected by arithmetic. It is also the only one of the three that ran Muse Code 1.1.1 rather than 1.2.1 - see the mean row's note.", "caveat": "One run at this tier. *** THIS CAVEAT SAID THE CEILING WAS FLAT AND IT IS NOT - CORRECTED 2026-09-13 *** It was written off the magnitude probe, which could not tell high, xhigh and max apart in thinking volume at n=3 on one fixed prompt. Both rungs below have since been run on the real benchmark: high scores 19 of 105 and xhigh scores 20, against this row's 33. So max is worth about thirteen points over either neighbour and is a purchase that pays on this stack. Superseded wording, kept verbatim rather than deleted: the effort ceiling is real but FLAT, the magnitude probe separates minimal, low and medium into non-overlapping bands but cannot tell high, xhigh and max apart at n=3, so max is the top of the dial and not a rung above high; read the tier as a ceiling that exists rather than as a purchase that pays. What SURVIVES from the probe is the grouping of high with xhigh, which the bench confirms on score as well as on tokens; what failed was extending that group upward to max. THE HARNESS VERSION IS CAPTURED, NOT TYPED: this row records Muse Code 1.1.1-R2514.1, read from muse --version at run time, because muse SELF-UPDATES - it replaced itself between the two legs of the sibling arm on 2026-09-12 and shipped a wrong version string to this board, which is a mistake a hand-written note can make twice and a captured value cannot make once. THE SHIM IN THE PATH IS A METER AND NOT A ROUTE: Muse Code's JSON stream carries no usage or token payloads at all, so without it this row would have no billing record and this board does not publish a cost column it cannot source. Pointed at Meta with the model rewrite and the provider pin both OFF it forwards the turn untouched - no translation of the conversation, the tools, the effort field or the stream - and only counts. It is the same instrument the OpenRouter row used, so the pair differs in ROUTE rather than in apparatus. CREDENTIAL: the subscription credential muse login stores in its own auth file, read at run time rather than copied, so a token muse refreshes cannot leave this arm on a stale one. billing=LIST ESTIMATE, NOT A BILL: a subscription account exposes no per-run credits delta to read, so this is costed from the run's own token counts at the same Meta list rates the OpenRouter row was billed at (1.25 in / 0.15 cache-read / 4.25 out per MTok), which keeps the two Muse Code rows comparable and makes neither a market price. Muse Code's reminder-observer side agent means the turn count and the token bill both include work no Claude Code row on this board does, so the cost column is not comparable to a Claude Code row of the same model even when the model is identical. The benchmark repos were re-installed from the same lockfile inside Linux before the agent started, because the Windows install lays down win32 binaries Linux node cannot execute; repo 2 needed --legacy-peer-deps and the exit code is recorded in the row.", "note": "Meta's OWN agent driving Meta's OWN model on Meta's OWN endpoint, at the top of the tier list that endpoint serves - no aggregator anywhere in the path. 33 of 105, which is the best Meta result on this board by a wide margin and nearly DOUBLE the 18/105 the same model and the same harness scored through OpenRouter a day earlier. (33 is the strict-match count this board publishes; the partial-credit figure is 34.5, because repo 2 contributed 3 partials at half each.) *** THE ATTRIBUTION IS NOW SETTLED, AND THIS NOTE ORIGINALLY GUESSED IT WRONG *** It first said the doubling could not be credited to the effort dial, because the route and the tier had moved together and the single-variable leg did not exist. That leg now exists and it runs the other way. Two facts arrived on 2026-09-13: the OpenRouter sibling was found to have run at high rather than at a dropped default, and a first-party leg at high scored 19/105. So the three rows differ one variable at a time - OpenRouter at high 18, Meta first-party at high 19, Meta first-party at max 33 - and THE ROUTE IS WORTH ABOUT A POINT WHILE THE TIER IS WORTH FOURTEEN. The caution was still right to publish: it was the honest state of the evidence, and the GLM-5.3 control it cited (the same move worth zero) remains true of GLM-5.3. Effort dials are not worth the same amount on every stack, which is why this board measures them per path instead of assuming. *** READ THE EFFORT LABEL AS A CLAIM ABOUT THE MAIN LOOP, NOT ABOUT THE RUN *** Receipt: effort-dial-probes/20260912-musecode-cli-effort-vs-wire.txt, which records every request body a run puts on the wire. Muse Code fires a reminder-observer side agent at efforts this flag does not control - fixed low, high and medium, sitting exactly where they were when the flag moved from minimal to max - so on a three-step probe at max, 4 of 10 calls carried max and six did not. The tier below is asserted, validated and echoed FOR THE MAIN LOOP, which is the agent that does the work; it is not true of every call this row is billed for. No other row on this board needed that distinction, because no other harness here runs a second agent. *** max IS THE CEILING, AND ultra IS A TRAP THIS ROW WALKED INTO AND BACKED OUT OF *** Receipt: effort-dial-probes/20260912-musespark13-meta-firstparty-effort.txt. Meta's endpoint validates reasoning.effort in TWO layers - a schema enum, then a per-model support list - and ECHOES the applied tier back inside the response object, which is stronger than any other first-party endpoint here manages: the others prove they READ the field by refusing bad values, this one also reports what it DID with a good one. minimal, low, medium, high, xhigh and max return 200 and echo themselves; none passes the schema and is then refused for this model; ultra and an invented bogus_zzz are refused as unknown variants; a bad model id fails differently again, as a 404. Omitting the field echoes high, so high is the measured default. Muse Code's own help offers ultra, and an earlier version of that receipt concluded a run launched at ultra would have ERRORED rather than run higher. It would not. It runs: the CLI prints that the gate ultra_reasoning_effort is closed and it is using xhigh, exits 0, and puts xhigh on the wire - ONE TIER BELOW max - with nothing in its JSON stream, its exit code or the response to say so. The endpoint never sees ultra because the CLI never sends it. An API's accept-list does not tell you what a CLI sends, any more than a CLI's accept-list tells you what an endpoint serves. *** AND THE DIAL IS NOW MEASURED, NOT JUST VALIDATED - IT BINDS AT THE BOTTOM AND SATURATES AT THE TOP *** Receipt: effort-dial-probes/20260913-musespark13-meta-effort-magnitude.txt, n=3 per tier on one fixed prompt, 18 calls, every call echoing the tier it was asked for. Mean reasoning tokens run 817 (minimal), 2238 (low), 5774 (medium), 9385 (high), 10757 (xhigh), 12655 (max) - a 15.5x bottom-to-top ratio against a 2.19x worst within-tier spread, so the dial is real by this board's magnitude rule. On that toy prompt minimal, low and medium sat in bands that touched nothing while high, xhigh and max overlapped almost entirely, and the receipt concluded max buys nothing measurable over high. *** THE BENCH SAYS OTHERWISE AND THE PROBE WAS THE THING THAT WAS WRONG *** A first-party leg at high scores 19/105 against this row's 33, and on the real workload the tiers separate hard: 178,300 reasoning tokens at high against 286,242 here, a 61% gap the fixed-prompt sweep could not see. The xhigh leg has since run as well and lands at 20, which locates the error precisely: the top of this dial is TWO rungs rather than one, high and xhigh within a point of each other and max thirteen above both, so the probe grouped correctly at high/xhigh and wrongly pulled max in with them. So a token-volume probe shows whether a dial is CONNECTED and not what it is WORTH, and the two must not be used interchangeably - which is also a caution against reading the Qwen3.8 saturation findings as statements about score, since no bench pair was ever run to test them. Honesty profile: 14 of 45 on repo 1 with no partials, 19 of 60 plus 3 partials on repo 2, ZERO claimed-only on either repo across three Meta rows running, and 16 genuine unplanted extras.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 27, 30, 33, 34, 39, 43, 44, 45, 47, 49, 50, 51, 60, 62, 69, 71, 76, 77, 83, 84, 87, 91, 92, 97, 98, 99, 102 ], "turns": 358, "turns_repo1": 184, "turns_repo2": 174, "group_rank": 200 }, { "id": "Gemini 4 Argon (high effort)", "n_runs": 1, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Gemini CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-10-13", "color": "#2C6FE0", "fixed": 33, "repo1_fixed": 15, "repo2_fixed": 18, "partial": 1, "claimed_only": 3, "extras": 7, "false_positive_fixes": 0, "wall_min": 85.1, "cost_usd": 32.08, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "One run (33 of 105). Run through Gemini CLI because Antigravity does not offer the model to this account. Cost is a list estimate; only 69% of prompt tokens were cached.", "note": "Google's Gemini 4 Argon at thinking level high - its top tier - through Gemini CLI on the first-party Gemini API. ONE RUN: 33 of 105 - 15 of 45 on repo 1 and 18 of 60 on repo 2 - with 1 partial fix(es), 3 claimed-only and 7 genuine unplanted extras; 85.1 minutes, $32.08. Google's previous best row here is Gemini 3.7 Flash (high) at 22. *** WHY GEMINI CLI, NOT ANTIGRAVITY *** Google's current CLI does not list Argon for the account used here and refuses an unlisted model before sending anything; the Gemini API serves it as gemini-4-argon. Gemini CLI 0.63.0 reached it directly. *** EFFORT high IS THE CEILING *** The API accepts thinkingLevel low, medium and high for this model, refuses minimal for it, and rejects max and ultra as invalid values; omitted means medium. Gemini CLI does not send a level for this model, so a local gateway pins high on every request and reads it back: high on all of them, served model gemini-4-argon on all of them. One-prompt magnitude only (medium 79 / high 408 thought tokens). *** COST *** a list estimate at Google's launch rate ($2 input / $0.10 cached input / $10 output per million tokens, thinking billed as output) from the API's own per-call usage. Only 69% of prompt tokens were cache reads, against the high 90s typical on this board, which is most of why it costs what it does. *** RECEIPTS *** Both legs exited 0, one after the other. Repo 2 met Google's spend-rate limit for a new account a few times and was slowed, not stopped. 1 of the 2 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 30, 33, 34, 39, 43, 45, 47, 49, 50, 51, 60, 69, 70, 71, 76, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 976, "turns_repo1": 434, "turns_repo2": 542, "group_rank": 2 }, { "id": "Fable 5.1", "n_runs": 1, "model": "Fable 5.1", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-01", "color": "#D9542B", "fixed": 33, "repo1_fixed": 15, "repo2_fixed": 18, "partial": 2, "claimed_only": 0, "extras": 7, "false_positive_fixes": 0, "wall_min": 36.3, "cost_usd": 48.58, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 22, 27, 28, 29, 30, 33, 34, 36, 39, 41, 43, 45, 47, 49, 50, 60, 70, 71, 76, 77, 82, 83, 84, 91, 92, 97, 98, 99, 100, 102 ], "turns": 308, "turns_repo1": 136, "turns_repo2": 172, "group_rank": 2 }, { "id": "Muse Spark 1.3 (max effort, Muse Code, Meta API) - mean of 5", "n_runs": 5, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-17", "color": "#E57EA2", "fixed": 32.2, "repo1_fixed": 14.2, "repo2_fixed": 18, "partial": 1.6, "claimed_only": 1.2, "extras": 24, "false_positive_fixes": 0, "wall_min": 85.6, "cost_usd": 18.11, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "FIVE RUNS, RANGE 6. The spread is the finding, not the mean: at n=3 this arm looked tight (range 2), and two more runs tripled the error bar. Treat 32.2 as an estimate with a wide known spread. TWO OF THE FIVE RUNS ARE SPLIT-VERSION - the harness updated itself mid-experiment and their repo-2 legs ran a newer build; the row explains why the evidence says that is not what lowered them. The replication is UNVERIFIED: this harness reports no context accounting, so nothing confirms the five runs shared a configuration beyond identical arm definitions. Repo figures round independently of the combined mean and can sum to 0.1 away from it.", "note": "THE DEEPEST REPLICATION ON THIS BOARD: FIVE RUNS OF ONE CONFIGURATION. Runs scored 33, 33, 35, 29 and 31 of 105 (repo 1: 14, 17, 15, 12, 13 - repo 2: 19, 16, 20, 17, 18). Mean 32.2, range 6. *** WHAT THE TWO EXTRA RUNS BOUGHT WAS AN ERROR BAR, NOT A SCORE. *** The mean moved 1.5 points, from 33.7 to 32.2. The RANGE went from 2 to 6. At n=3 this arm read as tight and well-behaved - 33, 33, 35. At n=5 it spans 29 to 35, which is wider than the gap between several adjacent rows here. The three-run version did not merely estimate the mean imprecisely; it understated how much this configuration varies, in the direction that made it look steadier than it is. Read 32.2 as an estimate with a known and fairly wide spread. *** THE HARNESS UPDATED ITSELF DURING THIS EXPERIMENT, AND TWO RUNS ARE SPLIT-VERSION. *** Muse Code polls a release channel and replaces its own binary. Run 1 ran Muse Code 1.1.1; runs 2-5 ran 1.2.1; and the repo-2 legs of runs 4 and 5 came up on 1.3.0. So those two runs were graded across two harness versions. There is no version pin in the launcher, the channel serves only the current release, and the previous binary is deleted on upgrade, so 1.2.1 cannot be restored and this is not repairable by re-running. Further drift is now suppressed. *** THE DRIFT IS NOT WHAT LOWERED THE SCORE, AND THE SPLIT IS WHAT SHOWS IT. *** The obvious worry is that runs 4 and 5 are low because 1.3.0 is worse. Two facts say otherwise. First, the drop lives on repo 1, where the harness never changed: all four of runs 2-5 ran 1.2.1 there and scored 17, 15, 12 and 13 - a spread of 5 on a single repo with the version held constant. Second, repo 2 shows no version effect at all: the 1.2.1 runs scored 16 and 20, and the two 1.3.0 runs scored 17 and 18, sitting inside that range rather than below it. The low runs are run-to-run variance. That the drift landed on the half where it cannot be doing the damage is luck, not design, and it is disclosed here rather than quietly dropped because the answer happened to be convenient. *** THE REGIME BEHIND THESE RUNS IS UNVERIFIED, NOT VERIFIED. *** CANNOT TELL (gbvb); CANNOT TELL (abhb) - the automated check reads context accounting out of transcripts and this harness reports none, so it cannot confirm the five runs shared a regime. Stated plainly because the alternative is implying a check that did not happen. *** WALL TIME IS AN UPPER BOUND AND IS NOT COMPARABLE ACROSS THE FIVE RUNS. *** Runs 1-3 ran their legs alone. Runs 4 and 5 were run concurrently with each other, a deliberate exception to the usual one-leg-per-provider rule, and their repo-2 legs also shared the machine with other work; every affected row says so itself. Contention can inflate a wall figure and cannot shrink it. Scores, tokens and cost are unaffected. *** THE FAILURE RATE IS PART OF THE PICTURE AND IS VISIBLE IN THE RECEIPTS. *** Runs 4 and 5 needed several attempts on repo 2. Three attempts died without producing a report because the account's subscription quota was exhausted mid-leg - the retry layer absorbed the rate-limit responses and the harness reported three different and all misleading causes, including two stream-idle timeouts and a suspected local network fault. Those attempts produced no report and no score, are NOT counted as low runs here, and cost 8.35 dollars between them. A leg that never finished is not a result. The five runs that did finish cost 90.55 dollars in total.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 38, 39, 40, 41, 43, 44, 45, 47, 49, 50, 51, 53, 56, 57, 60, 62, 64, 69, 70, 71, 76, 77, 80, 82, 83, 84, 87, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 4, "20": 5, "21": 4, "22": 4, "23": 3, "25": 4, "26": 3, "27": 4, "28": 1, "29": 4, "30": 5, "33": 4, "34": 4, "38": 1, "39": 5, "40": 1, "41": 1, "43": 5, "44": 2, "45": 4, "47": 4, "49": 4, "50": 5, "51": 2, "53": 1, "56": 1, "57": 1, "60": 4, "62": 5, "64": 1, "69": 2, "70": 2, "71": 5, "76": 5, "77": 5, "80": 1, "82": 2, "83": 5, "84": 5, "87": 1, "91": 1, "92": 5, "95": 1, "97": 5, "98": 5, "99": 5, "100": 2, "102": 5 }, "coverage_runs": 5, "turns": 450, "turns_repo1": 169, "turns_repo2": 281, "group_rank": 200 }, { "id": "Union Alpha (stealth; free)", "n_runs": 1, "model": "Pareto (ex-Union Alpha)", "vendor": "Unbiased", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-17", "color": "#EC13EC", "fixed": 32, "repo1_fixed": 16, "repo2_fixed": 16, "partial": 2, "claimed_only": 3, "extras": 11, "false_positive_fixes": 0, "wall_min": 32.7, "cost_usd": 3.3, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-17 by the n=3 MEAN row. Nothing here is retracted: the 32 happened and this is its receipt, judged twice and agreed on bug-for-bug. Kept rather than deleted because a supersession is only legible next to the row it supersedes - and this one sat MID-RANGE of the three runs (32, against 34 and 26) yet still above the 30.7 mean, so a reader comparing against 32 was comparing against a slightly lucky draw rather than an outlier.", "caveat": null, "note": "REVEALED 2026-09-17, hours after this row went up: the stealth slug was unbiased/pareto. OpenRouter announced it directly, and said community detective work cut the stealth period short. The endpoint then returned 404 and every stealth slug left the model list. The row keeps the stealth name because the stealth endpoint is what was measured, and it now carries its real vendor. *** IT TIES GPT-5.6 Terra (max effort) EXACTLY: 16 of 45 and 16 of 60, the same split, for nothing in 32.7 minutes against $27.98 in 159.7. The tie is on the number and not on the work - the two runs share 23 of their 32 bugs, so nine on each side are bugs the other missed. *** PARETO IS NOT A MODEL, IT IS A COMPOSITE - AND THAT IS THE WHOLE STORY OF THIS ROW. Unbiased describes it as running several frontier and open-source models in parallel on every request and selecting or synthesising the output rather than routing to one. That is measurable, and it was measured twice. BEFORE the reveal, from readings taken while the slug was live: a prefix-free doubling probe returned 160 and 200 tokens for one fixed 437-character passage - a 40-token gap where the entire spread across ten different labs on that same passage is 82 tokens, with the hidden prefix cancelled by construction and no special-token markers present for a stripping state to explain. One model has one vocabulary, so this had to be more than one backend. AFTER the reveal, against the live endpoint: the same passage returns 138, 160 AND 200, and 8 of 10 corpus passages come back with two or three different counts. The prediction and the confirmation both hold. WHAT THE WIRE WORK GOT RIGHT AND WRONG: the exclusions were sound as far as they went - Anthropic, xAI, Google and Meta are not the developer - but the question was malformed, because the developer is an aggregator and the weights belong to whoever sits on the panel for that call. An earlier version of this note leaned toward OpenAI on residual evidence. That lean was withdrawn on 2026-09-17, a day before the reveal, and the reveal says it was answering the wrong question rather than answering it wrongly. READ THE TOKEN COLUMNS AS A SELF-REPORT AND NOTHING FIRMER - more so now than when that caveat was written, because the counter behind them belongs to whichever panel member answered: the same fixed string, cache-busted, came back at about 200 prompt tokens with cached_tokens=0 or at 169-172 with cached_tokens=12. THE RUN WAS GENUINELY FREE AND WHAT IT MEASURED WAS EXACTLY ZERO, but the preview has ended: unbiased/pareto now lists at $2.50 per million prompt, $7.50 completion, $0.25 cached read. Repricing this run's own measured token counts at those rates gives $3.30 - against the $27.98 that GPT-5.6 Terra (max effort) actually billed for the identical 16+16 split. So the cell reads $3.30 and is marked cost_list, which the glossary defines as a token-count estimate at published list rates and not a bill. It was $0.00 until the preview ended; an honest zero then stopped supporting the only comparison the column exists for, so the figure was repriced generator-side. Nothing was actually spent, and the published receipts still say so. The comparison survives the preview ending either way, at 8.5x cheaper instead of free. *** THE SLUG SERVED MORE THAN ONE VOCABULARY. *** Predicted before the reveal and confirmed against the revealed endpoint: at least three backends answer to this slug. The benchmark runs themselves do NOT look mixed: the three replicates agree at mean Jaccard 0.628 on their fixed-bug sets against a board median of 0.632 over 24 replicate families, which is ordinary. The probe evidence suggests the router was length-sensitive, so short probes and long agentic runs need not have landed on the same backend. The score is what was measured; that it describes a single model is not something this board can claim. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, AND WHAT THAT COST WAS MEASURED RATHER THAN ASSERTED. A model may not judge itself or a sibling; with the lab unidentified the routing could not be read off a vendor at all, so the scoring pass ran on grok-4.6. The reveal vindicates that more strongly than the reasoning given for it: a composite has no single vendor to check a judge against, and third-party analysis of the panel places GPT-6 Astra on the escalation path - the default judge's own family, inside the thing being judged. Every other row here was judged by codex, and the measured judge-to-judge spread on one transcript is about 2 points of 105, so this run was ALSO judged by codex into a separate scratch pass that could not redefine the scoring one. The two judges selected the SAME 32 bugs - identical sets on both repos, not merely equal counts. The number does not depend on who read it. *** THERE IS NO EFFORT DIAL ON THIS ROUTE. supported_parameters carries no reasoning field of any kind, the model returns reasoning_tokens=0 with empty reasoning_details on a prompt that makes every reasoning model think, and an invented tier is refused by the gateway in 0.2s without ever reaching the provider. So effort is 'default' and deliberately not 'inert_default': nothing was asserted, and there was no dial to probe. *** ONE RUN, AND THE ROUTE IS VISIBLY NOISY. Six byte-identical temperature-0 calls came back bimodal - about 8s and about 32s, 20 to 218 completion tokens, a 10.9x spread inside one condition. Two replicates are running; when they land, their mean supersedes this row.", "fixed_bugs": [ 13, 16, 20, 25, 29, 30, 33, 34, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 70, 76, 77, 78, 80, 82, 83, 84, 91, 92, 96, 100, 102 ], "turns": 89, "turns_repo1": 46, "turns_repo2": 43, "group_rank": 6 }, { "id": "Gemini 4 Argon (low effort, Antigravity CLI)", "n_runs": 1, "model": "Gemini 4 Argon", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "low", "effort_status": "verified", "date": "2026-10-14", "color": "#2C6FE0", "fixed": 32, "repo1_fixed": 12, "repo2_fixed": 20, "partial": 1, "claimed_only": 0, "extras": 3, "false_positive_fixes": 0, "wall_min": 44.9, "cost_usd": 5.97, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-10-14 by the mean of three runs (35.3 of 105). This row reported 32 of 105 from a single run (run 2, the only one complete at the time). Nothing here is retracted - it is the receipt for what the board showed before runs 1 and 3 finished.", "caveat": "One run - the other two low runs did not finish their repo-2 legs. Judged by Grok, not Codex like the other Argon rows. Cost is a list estimate from Google's per-call usage.", "note": "Google's Gemini 4 Argon at thinking level low - the lowest tier the API accepts for it - through the Antigravity CLI, set up exactly as the medium and high rows. ONE RUN: 32 of 105 - 12 of 45 on repo 1 and 20 of 60 on repo 2 - with 1 partial fix(es), 0 claimed-only and 3 genuine unplanted extras; 44.9 minutes, $5.97. Medium stands at 39.3 and high at 40.7 (means of three); low is the cheapest tier by a little, not by much. *** WHY ONE RUN *** Three low runs were started. In runs 1 and 3 the repo-2 leg did not finish: twice the local gateway stopped answering before Google started its stream (the CLI gives up on a response that has not begun within about a minute), and the resumed conversations then hit the API key's depleted prepaid credit. Run 2 completed both repos and is the row. *** A DIFFERENT JUDGE *** this row was judged by Grok (grok-4.7), not Codex gpt-5.5 like the other Argon rows: from about 16:56 on Oct 14 Codex refused gpt-5.5 for the account. Grok is this board's second calibrated judge and is not a sibling of Gemini, but scores from two judges are not perfectly interchangeable. *** EFFORT low *** agy sends thinkingLevel LOW for gemini-4-argon-low, read back at the gateway; a couple of agy's own side calls per run go out at HIGH regardless. *** COST *** a list estimate from Google's per-call usage at its launch rate ($2 input / $0.10 cached input / $10 output per million tokens), split between legs by agy's per-leg share. *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 23, 26, 30, 33, 34, 39, 43, 45, 47, 50, 60, 62, 70, 76, 77, 79, 82, 83, 84, 91, 92, 93, 94, 97, 98, 99, 100, 102 ], "turns": 462, "turns_repo1": 130, "turns_repo2": 332, "group_rank": 4 }, { "id": "GPT-6 Sol (max effort)", "n_runs": 1, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 32, "repo1_fixed": 18, "repo2_fixed": 14, "partial": 1, "claimed_only": 0, "extras": 48, "false_positive_fixes": 0, "wall_min": 60.4, "cost_usd": 10.03, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (31.0 of 105). This row reported 32 of 105 from a single run, published the night the model shipped. The replicates are byte-identical re-runs; across 2 the runs span 30 to 32. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean, on a model measured the night it shipped - and this board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat 32 as one draw. It also runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; the harness is therefore not held fixed across that comparison. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The wall figure is contended.", "note": "OpenAI's GPT-6 Sol at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 32 of 105: 18 of 45 on repo 1 and 14 of 60 on repo 2, 48 genuine extras, 60.4 minutes, $10.03. ONE RUN, measured the night the model shipped. *** THE NEWER CHECKPOINT DOES NOT LEAD ITS OWN FAMILY HERE *** GPT-6 Astra publishes 45 of 105 at max (mean of 3) and 43 at xhigh on this same benchmark; GPT-5.6 Sol publishes 43.5 at max (mean of 2). This row lands at 32, below all three, at a fraction of the cost - $10 against GPT-5.6 Sol's $95. A single run cannot settle a gap that size against a mean, but it is a large gap in the direction nobody would have guessed from the release order. *** IT KILLED AN ALL-TIME SURVIVOR *** one repo-1 bug, A6, had outlived every model in every scored run on this board until now - a settings-persistence ordering defect, where a preference was written to storage before the change it described had actually taken effect, so a switch that failed or was dismissed left the stored value and the running session disagreeing. The survivor count drops from 32 to 31. That is worth separating from the score: this row sits in the middle of the table and still opened ground nothing above it had touched. It is also a variance event, not a capability - survivor kills on this board have not reproduced on demand in later runs. *** IT FINDS A LOT OF THINGS NOBODY PLANTED *** 48 genuine extras - real defects in the repos that are not part of the planted set - is among the highest counts on this board, from a row in the middle of the scoring table. Extras are reported separately and never folded into the score, but a model that flags this many unplanted defects while fixing fewer planted ones is spending its attention somewhere the planted total cannot see. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows; the choice was a one-version harness delta or no row at all. Read the comparison above with that in it. (The rejection message is generic - a deliberately fake slug returns it verbatim - so it is not evidence about any model's existence.) *** THE FIRST TWO ATTEMPTS AT THIS ROW WERE VOID, AND THE REASON IS WORTH KNOWING *** the arm was first pinned to npm's shim rather than the real executable. On Windows that shim resolves to a .cmd, running a .cmd makes the shell re-parse the whole argument list, and a NEWLINE ENDS THE COMMAND - so the model received the benchmark prompt's first line, the repo description, and none of the numbered instructions. Both legs launched, billed and exited 0 with nothing to show the harness: one edited code and wrote no report it had never been asked for, the other replied asking what to work on. Nothing in the leg's own output said the prompt was short. Those legs are kept in the ledger, voided and labelled. This row is a clean re-run of both repos afterwards. A third earlier attempt died on an unrelated fault in the CLI's sandbox setup, since fixed. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Both legs were judged by grok-4.7 (the id is the one the scoring receipt records as SERVED, not the one the judge config asked for - from 2026-09-22 the endpoint has been resolving a request for grok-4.5 to grok-4.7, and notes published before 2026-09-23 named the requested id), blinded to the arm's identity. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The run shared the laptop with legs from another vendor, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 6, 13, 16, 20, 21, 22, 25, 27, 29, 30, 33, 34, 37, 39, 40, 43, 44, 45, 47, 49, 50, 62, 71, 76, 77, 82, 83, 84, 98, 99, 100, 102 ], "turns": 365, "turns_repo1": 154, "turns_repo2": 211, "group_rank": 200 }, { "id": "Sonnet 5.5 (high effort)", "n_runs": 1, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 32, "repo1_fixed": 15, "repo2_fixed": 17, "partial": 1, "claimed_only": 3, "extras": 12, "false_positive_fixes": 0, "wall_min": 24.5, "cost_usd": 16.73, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-29 by the mean of 3 (34.3 of 105). This row reported 32 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 32 to 37. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat this as one draw. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at high effort through Claude Code on the first-party API, measured the evening it shipped. 32 of 105: 15 of 45 on repo 1 and 17 of 60 on repo 2, 12 genuine extras, 24.5 minutes, $16.73. ONE RUN. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** HOW IT SPENT THE RUN *** On repo 1 it took 138 turns (the harness's own count); it first wrote its report 16 minutes in and its last model call came at minute 16. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 156 turns (the harness's own count); it first wrote its report 8 minutes in and its last model call came at minute 8. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 511,985 tokens; 0 compactions across 2 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 27, 28, 29, 30, 33, 34, 39, 43, 45, 47, 49, 50, 60, 61, 70, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100 ], "turns": 294, "turns_repo1": 138, "turns_repo2": 156, "group_rank": 502 }, { "id": "Opus 5.5 (high effort) - mean of 2", "n_runs": 2, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 32, "repo1_fixed": 13.5, "repo2_fixed": 18.5, "partial": 2.5, "claimed_only": 1, "extras": 9.5, "false_positive_fixes": 0, "wall_min": 24.1, "cost_usd": 22.4, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (31.7 of 105). This row reported 32.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 31 to 32. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 32 and 32 of 105 - spread 0. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at high effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 32 and 32 of 105 (repo 1: 14, 13 of 45 - repo 2: 18, 19 of 60). Mean 32.0, spread 0. *** THIS SUPERSEDES OPUS 5.5 (HIGH EFFORT) (32 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 23.8, 24.4 minutes, mean 24.1; cost $23.03, $21.76, mean $22.40. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 7 and 12 across the runs, mean 9.5 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 25, 26, 29, 30, 33, 34, 37, 39, 43, 45, 47, 49, 50, 60, 62, 71, 76, 77, 80, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 2, "17": 1, "20": 2, "21": 1, "22": 2, "25": 1, "26": 1, "29": 2, "30": 2, "33": 2, "34": 2, "37": 2, "39": 2, "43": 2, "45": 2, "47": 2, "49": 2, "50": 2, "60": 2, "62": 2, "71": 2, "76": 2, "77": 2, "80": 1, "82": 1, "83": 2, "84": 2, "91": 2, "92": 2, "95": 2, "97": 1, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 266, "turns_repo1": 97, "turns_repo2": 168, "group_rank": 202 }, { "id": "Opus 5.5 (high effort)", "n_runs": 1, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-22", "color": "#DE6C48", "fixed": 32, "repo1_fixed": 14, "repo2_fixed": 18, "partial": 3, "claimed_only": 1, "extras": 7, "false_positive_fixes": 0, "wall_min": 23.8, "cost_usd": 23.03, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (32.0 of 105). This row reported 32 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 32 to 32. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean. This is the high rung of a four-rung same-day sweep (medium 30, high 32, xhigh 35, max 43 of 105); the ladder is monotone, but no gap in it exceeds the 13-point run-to-run spread this board has measured on a single identical configuration, so read the direction, not the ordering of adjacent rungs. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. The wall figure is contended.", "note": "Anthropic's Opus 5.5 at high effort through Claude Code on the first-party API. 32 of 105: 14 of 45 on repo 1 and 18 of 60 on repo 2, 7 genuine extras, 23.8 minutes, $23.03. ONE RUN. *** THIS IS ONE RUNG OF A FOUR-RUNG SWEEP *** All four Opus 5.5 rows on this board are the same arm definition with one character changed - the effort dial - run on the same day, on the same machine, against the same two repos, judged the same way. The ladder is monotone: medium 30, high 32, xhigh 35, max 43 of 105, costing $16.55, $23.03, $35.14 and $60.49. *** BUT READ THE LADDER AS A TREND, NOT AS FOUR LEVELS *** every rung is a single run, and this board has already measured a 13-POINT spread between two runs of one identical max-effort configuration elsewhere. No gap on this ladder is that wide - the largest, max over xhigh, is 8. So the direction is credible across four points; the ordering of any two adjacent rungs is not resolved at n=1, and nothing here says high is reliably worse than the rung above it. *** THE DIAL BINDS - MEASURED, NOT ASSUMED *** this board has caught several third-party routes that accept a reasoning field and ignore it, so 'first party' is not taken on trust. Token spend on repo 2 rises monotonically with the dial across the sweep - 33.4M at medium, 46.8M at high, 82.7M at xhigh, 150.2M at max, a 4.5x span - and wall rises with it. This rung burned 46.8M. Whatever else is true, the setting is reaching the model and changing how hard it works. *** THE CONTEXT WINDOW, WHICH IS WHY THESE ROWS DO NOT COMPARE TO THE OPUS 5 LADDER *** this arm runs the model string claude-opus-5-5[1m] and therefore a 1M window: it held a peak of 482,838 tokens on repo 2 and crossed ZERO compaction boundaries, carrying the repo whole through every turn. Every Opus 5 row on this board runs a bare slug, which the CLI treats as 200K, and those runs compacted repeatedly mid-task. Comparing this ladder to the Opus 5 ladder moves two variables at once - model version AND how much of the codebase the model could hold while reasoning about it. The [1m] suffix was verified on CLI 2.1.280 rather than assumed (an earlier arm on this board requested a 1M window and was silently normalised to the bare slug on 2.1.272); init.model returns 'claude-opus-5-5[1m]' verbatim. Autocompact ran at the CLI's own default - a user-level CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=30 that had been reaching legs on this machine was removed before the sweep and its absence re-verified in a leg-shaped probe. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Opus 5.5 rates: $4 input, $20 output, $0.20 cache read - a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x - and $8 cache write, the 1-hour rate, which is what this harness actually writes. Not an invoice, hence `list`. *** RECEIPTS *** Both legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The run shared the laptop with benchmark legs from another vendor, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 29, 30, 33, 34, 37, 39, 43, 45, 47, 49, 50, 60, 62, 71, 76, 77, 80, 83, 84, 91, 92, 95, 98, 99, 100, 102 ], "turns": 269, "turns_repo1": 107, "turns_repo2": 162, "group_rank": 202 }, { "id": "GPT-5.6 Terra (max effort)", "n_runs": 1, "model": "GPT-5.6 Terra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-08-27", "color": "#69BFA6", "fixed": 32, "repo1_fixed": 16, "repo2_fixed": 16, "partial": 1, "claimed_only": 0, "extras": 45, "false_positive_fixes": 0, "wall_min": 159.7, "cost_usd": 27.98, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "The $2/Mtok mid tier at max: 32/105, just below Luna max (33) and above Grok xhigh / Opus max (27) at a fraction of Sol's cost. 45 genuine unplanted extras, the bug-max signature.", "fixed_bugs": [ 16, 20, 22, 25, 28, 29, 30, 33, 34, 35, 37, 39, 40, 43, 44, 45, 46, 47, 49, 50, 62, 71, 76, 77, 80, 83, 84, 92, 97, 98, 99, 102 ], "turns": 875, "turns_repo1": 389, "turns_repo2": 486, "group_rank": 500 }, { "id": "Opus 5.5 (high effort) - mean of 3", "n_runs": 3, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 31.7, "repo1_fixed": 13, "repo2_fixed": 18.7, "partial": 2.3, "claimed_only": 1.4, "extras": 10.7, "false_positive_fixes": 0, "wall_min": 23.9, "cost_usd": 22.25, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 32, 32 and 31 of 105 - spread 1. Read the mean with that width, not as a point. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at high effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 32, 32 and 31 of 105 (repo 1: 14, 13, 12 of 45 - repo 2: 18, 19, 19 of 60). Mean 31.7, spread 1. *** THIS SUPERSEDES OPUS 5.5 (HIGH EFFORT) - MEAN OF 2 (32.0 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 23.8, 24.4, 23.7 minutes, mean 24.0; cost $23.03, $21.76, $21.94, mean $22.24. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 7, 12 and 13 across the runs, mean 10.7 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 25, 26, 29, 30, 33, 34, 37, 39, 43, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 80, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 3, "17": 1, "20": 3, "21": 1, "22": 3, "25": 2, "26": 1, "29": 3, "30": 3, "33": 3, "34": 3, "37": 2, "39": 3, "43": 3, "45": 3, "47": 3, "49": 3, "50": 3, "60": 3, "62": 3, "70": 1, "71": 3, "76": 3, "77": 3, "80": 1, "82": 2, "83": 3, "84": 3, "91": 2, "92": 3, "95": 3, "97": 2, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 267, "turns_repo1": 92, "turns_repo2": 176, "group_rank": 202 }, { "id": "Muse Wonder 1.0 (max effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Wonder 1.0", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "max", "effort_status": "first_party", "date": "2026-10-13", "color": "#DE5887", "fixed": 31.7, "repo1_fixed": 12, "repo2_fixed": 19.7, "partial": 3, "claimed_only": 1.6, "extras": 13.3, "false_positive_fixes": 0, "wall_min": 96.2, "cost_usd": 27.87, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Mean of three runs (30, 36 and 29 of 105). Effort max is the top tier Meta lists for the model, not magnitude-probed. Cost is a list estimate at Meta's published rates; the runs used a subscription.", "note": "Meta's Muse Wonder 1.0 at max effort through Muse Code, Meta's own agent CLI, released 2026-10-12. MEAN OF THREE RUNS of one configuration: 30, 36 and 29 of 105 (repo 1: 12, 13 and 11 of 45 - repo 2: 18, 23 and 18 of 60), mean 31.7. Genuine unplanted extras 5, 26 and 9; claimed-only fixes 2, 2 and 1. Walls 97.5, 110.6 and 80.6 minutes; cost $30.00, $35.57 and $18.04. *** AGAINST ITS PREDECESSOR *** Muse Spark 1.3 at max, same harness and route, stands at 32.2 as a mean of five (29 to 35). Wonder's mean of 31.7 sits inside that range: below Spark's repo-1 mean (14.2) in all three runs, above its repo-2 mean (18.0) on average, level overall. *** EFFORT max *** the top entry of the reasoning-effort list Meta's own console and Muse Code's model catalog give this model (minimal, low, medium, high, xhigh, max); Muse Code's `ultra` is a closed gate that silently sends xhigh, so max is the highest tier the harness sends. A first-party tier taken at the vendor's word, not magnitude-probed on this model. The tier describes Muse Code's main loop; its reminder-observer side agent runs at its own fixed efforts. *** COST *** a list estimate at the rates Meta's hosted endpoint lists for Wonder ($4 input / $0.20 cache read / $20 output per million tokens), from each leg's own token tally; the runs used a subscription, so there is no bill. 96.7% of prompt tokens were cache reads. *** RECEIPTS *** All six counted legs exited 0. Muse Code 1.4.1 (1.4.1-R4503.1). Run 1 took its repos one after the other; runs 2 and 3 took them at the same time. 4 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. 4 further legs ended on account errors before writing a report (one billing refusal at start, then the subscription's usage window); they say nothing about the model and are not counted. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 26, 27, 28, 29, 30, 37, 39, 43, 45, 47, 49, 50, 51, 60, 62, 66, 67, 69, 70, 71, 76, 77, 81, 82, 83, 84, 92, 93, 95, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 3, "25": 3, "26": 1, "27": 2, "28": 2, "29": 3, "30": 3, "37": 1, "39": 3, "43": 3, "45": 2, "47": 3, "49": 3, "50": 3, "51": 2, "60": 3, "62": 3, "66": 1, "67": 1, "69": 2, "70": 3, "71": 2, "76": 3, "77": 2, "81": 1, "82": 1, "83": 3, "84": 3, "92": 1, "93": 1, "95": 2, "96": 1, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 258, "turns_repo1": 128, "turns_repo2": 130, "group_rank": 100 }, { "id": "GPT-5.6 Luna (max effort) - mean of 2", "n_runs": 2, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-22", "color": "#79C6AF", "fixed": 31.5, "repo1_fixed": 16.5, "repo2_fixed": 15, "partial": 3, "claimed_only": 0, "extras": 44, "false_positive_fixes": 0, "wall_min": 170.0, "cost_usd": 2.82, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-22 by the mean of three (31.3 of 105). This row was the mean of two runs at 31.5, and it was featured. A third independent run - delayed by a Grok judge-bridge fault unrelated to Luna - completed the originally planned n=3, moving the mean to 31.3 across runs of 33, 30 and 31. Nothing here is retracted; it is kept as the receipt for what the board showed before this third run.", "caveat": "Two runs of an identical configuration scoring 33 and 30 of 105 - n=2, not the n=3 originally planned, because a third replicate's judging failed on a Grok bridge initialization error unrelated to Luna. Cost is a TOKEN ESTIMATE at list rates, not a bill, and is comparable only to other list-priced rows. Runs shared the laptop with benchmark legs from other vendors, so wall figures are contended and biased upward; scores and cost are unaffected.", "note": "GPT-5.6 Luna at max effort through OpenAI's own Codex CLI, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 33 and 30 of 105 (repo 1: 17, 16 of 45 - repo 2: 16, 14 of 60). Mean 31.5, spread 3. *** THIS SUPERSEDES A SINGLE RUN THAT WAS THE HIGHEST NUMBER ON THIS BOARD *** The earlier row said 33 of 105 from one draw, and it was FEATURED, which makes it the first number a reader meets. That is the most over-read kind of figure a benchmark can publish: nothing about it was false - it said n=1 on its face - but a top-of-board score from a single draw invites a confidence the sample cannot carry. *** WHY n=2 AND NOT n=3 *** A third replicate (gpt56lunamaxr3) was attempted and its two legs ran clean, but judging it failed identically 3 times per bench (`TimeoutError: ACP request timed out: initialize` in the Grok ACP judge bridge, before any real judging began) - a bridge/initialization problem, not a Luna problem, and not resolved before today's benchmarking window closed. This row is published honestly as n=2 rather than waiting on or faking a third value; r3 may be added as a further replicate later if the judge bridge issue is resolved. *** WALL AND COST *** Walls 85.8, 254.3 minutes, mean 170.1; cost $1.80, $3.85, mean $2.83. COST IS A TOKEN ESTIMATE at OpenAI's published rates, not an invoice - the Codex CLI on a subscription does not expose a per-run bill, so this column is `list` and comparable only to other list rows. *** GENUINE EXTRAS *** 31 and 57 across the two runs, mean 44.0 - real defects found in the repos that are not part of the planted set. Not scored, reported because a model that finds bugs nobody planted is doing something the planted total cannot see. *** THE TIER IS FIRST-PARTY AND THAT IS WHY IT CAN BE ASSERTED *** max is Luna's top tier on OpenAI's own endpoint, which validates and separates its effort levels rather than silently clamping an unknown string the way several third-party routes on this board do. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** A model may never judge itself or a sibling, and this is an OpenAI model, so the default Codex judge is disqualified. Two legs were judged by grok-4.5 and two by grok-4.7 (the id is the one the scoring receipt records as SERVED, not the one the judge config asked for - from 2026-09-22 the endpoint has been resolving a request for grok-4.5 to grok-4.7, and notes published before 2026-09-23 named the requested id), blinded to the arm's identity. The runs themselves shared the machine with legs from other vendors, which the row's wall marker discloses; scores, tokens and cost are untouched by that - only the clock is. *** RECEIPTS *** All four legs exited 0. Legs SEQUENTIAL within every run.", "fixed_bugs": [ 13, 16, 17, 20, 21, 22, 25, 27, 29, 30, 33, 34, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 62, 70, 71, 76, 77, 80, 83, 84, 91, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "17": 1, "20": 2, "21": 1, "22": 2, "25": 2, "27": 1, "29": 2, "30": 2, "33": 2, "34": 2, "37": 2, "39": 1, "40": 2, "41": 1, "42": 1, "43": 2, "44": 2, "45": 1, "46": 1, "47": 2, "49": 2, "50": 1, "62": 2, "70": 1, "71": 1, "76": 2, "77": 2, "80": 1, "83": 2, "84": 2, "91": 1, "92": 2, "97": 1, "98": 2, "99": 2, "100": 2, "102": 1 }, "coverage_runs": 2, "turns": 876, "turns_repo1": 404, "turns_repo2": 472, "group_rank": 600 }, { "id": "GPT-5.6 Luna (max effort) - mean of 3", "n_runs": 3, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-22", "color": "#79C6AF", "fixed": 31.3, "repo1_fixed": 16, "repo2_fixed": 15.3, "partial": 3.3, "claimed_only": 0, "extras": 54.7, "false_positive_fixes": 0, "wall_min": 187.1, "cost_usd": 3.15, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration spanning 30 to 33 of 105 - read the mean with that width in mind, not as a point. Cost is a TOKEN ESTIMATE at list rates, not a bill, and is comparable only to other list-priced rows. Runs shared the laptop with benchmark legs from other vendors, so wall figures are contended and biased upward; scores and cost are unaffected.", "note": "GPT-5.6 Luna at max effort through OpenAI's own Codex CLI, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 33, 30 and 31 of 105 (repo 1: 17, 16, 15 of 45 - repo 2: 16, 14, 16 of 60). Mean 31.3, spread 3. *** THIS SUPERSEDES THE MEAN OF TWO (31.5 of 105) *** a third independent run - held back earlier today by a Grok judge-bridge fault unrelated to Luna itself - completes the originally planned n=3. The mean moved from 31.5 to 31.3; the three runs span 30 to 33, and that span is the honest width of this measurement. *** WALL AND COST *** Walls 85.8, 254.3, 221.5 minutes, mean 187.2; cost $1.80, $3.85, $3.80, mean $3.15. COST IS A TOKEN ESTIMATE at OpenAI's published rates, not an invoice - the Codex CLI on a subscription does not expose a per-run bill, so this column is `list` and comparable only to other list rows. *** GENUINE EXTRAS *** 31, 57 and 76 across the three runs, mean 54.7 - real defects found in the repos that are not part of the planted set, rising noticeably across the replicates; reported because a model that finds bugs nobody planted is doing something the planted total cannot see, not folded into the scored total. *** THE TIER IS FIRST-PARTY AND THAT IS WHY IT CAN BE ASSERTED *** max is Luna's top tier on OpenAI's own endpoint, which validates and separates its effort levels rather than silently clamping an unknown string the way several third-party routes on this board do. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** A model may never judge itself or a sibling, and this is an OpenAI model, so the default Codex judge is disqualified - and so is any other first-party GPT-5.6 checkpoint on this board (e.g. Sol, Terra), since the rule is about shared weights, not identity-knowledge alone. Four legs were judged by grok-4.7 and two by grok-4.5 (the id is the one the scoring receipt records as SERVED, not the one the judge config asked for - from 2026-09-22 the endpoint has been resolving a request for grok-4.5 to grok-4.7, and notes published before 2026-09-23 named the requested id), blinded to the arm's identity. The runs themselves shared the machine with legs from other vendors, which the row's wall marker discloses; scores, tokens and cost are untouched by that - only the clock is. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run.", "fixed_bugs": [ 2, 13, 16, 17, 20, 21, 22, 25, 27, 29, 30, 33, 34, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 62, 70, 71, 76, 77, 80, 83, 84, 91, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "2": 1, "13": 3, "16": 3, "17": 2, "20": 3, "21": 1, "22": 3, "25": 3, "27": 1, "29": 3, "30": 3, "33": 3, "34": 3, "37": 3, "39": 1, "40": 2, "41": 2, "42": 1, "43": 3, "44": 3, "45": 1, "46": 2, "47": 3, "49": 3, "50": 1, "62": 3, "70": 2, "71": 2, "76": 3, "77": 3, "80": 2, "83": 3, "84": 3, "91": 1, "92": 3, "97": 2, "98": 3, "99": 3, "100": 3, "102": 1 }, "coverage_runs": 3, "turns": 990, "turns_repo1": 542, "turns_repo2": 448, "group_rank": 600 }, { "id": "GPT-6.1 Sol (low effort)", "n_runs": 1, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-09-29", "color": "#2BA580", "fixed": 31, "repo1_fixed": 15, "repo2_fixed": 16, "partial": 4, "claimed_only": 0, "extras": 32, "false_positive_fixes": 0, "wall_min": 36.5, "cost_usd": 1.77, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-10-01 by the mean of 2 (22.5 of 105). This row reported 31 of 105 from a single run. Its note announced two replicates; this tier was then cut to one, so the mean is of 2. The replicate is a byte-identical re-run; across 2 the runs span 14 to 31. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat 31 as one draw until the replicates land. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates (Codex caps the context for this model below 272K). Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at low effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 31 of 105: 15 of 45 on repo 1 and 16 of 60 on repo 2, 32 genuine extras, 36.5 minutes, $1.77, 91 turns (48 and 43). ONE RUN - two replicates follow and will supersede this row. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at medium 28, GPT-6.1 Sol at high 38, GPT-6.1 Sol at xhigh 41, GPT-6.1 Sol at max 44. At the same tier the board publishes GPT-6 Sol at low 6, GPT-6 Astra at low 27, Sonnet 5.5 at low 19 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at low. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 37, 40, 41, 43, 44, 45, 47, 49, 50, 62, 68, 69, 76, 77, 82, 83, 84, 92, 98, 99, 100, 102 ], "turns": 91, "turns_repo1": 48, "turns_repo2": 43, "group_rank": 104 }, { "id": "GPT-6 Sol (max effort) - mean of 2", "n_runs": 2, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 31, "repo1_fixed": 17, "repo2_fixed": 14, "partial": 1.5, "claimed_only": 0, "extras": 41, "false_positive_fixes": 0, "wall_min": 59.3, "cost_usd": 9.1, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (29.3 of 105). This row reported 31.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 26 to 32. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 32 and 30 of 105 - spread 2. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. It also runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; the harness is therefore not held fixed across that comparison. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. Wall figures are contended.", "note": "OpenAI's GPT-6 Sol at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. Published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 32 and 30 of 105 (repo 1: 18, 16 of 45 - repo 2: 14, 14 of 60). Mean 31.0, spread 2. *** THIS SUPERSEDES GPT-6 SOL (MAX EFFORT) (32 OF 105) *** *** THE NEWER CHECKPOINT STILL DOES NOT LEAD ITS OWN FAMILY *** GPT-6 Astra publishes 45 of 105 at max (mean of 3) and 43 at xhigh on this same benchmark; GPT-5.6 Sol publishes 43.5 at max (mean of 2). This row lands at 31.0, below all three, at a fraction of the cost. The single run that first showed this was one draw on a board that has measured a 13-point spread between two runs of one identical max-effort configuration - which is exactly why it was replicated rather than left to stand. *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string gpt-6-sol, same effort, same timeout, same pinned executable, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** IT FINDS A LOT OF THINGS NOBODY PLANTED *** 48 and 34 genuine extras across the runs, mean 41.0 - real defects in the repos that are not part of the planted set, among the highest counts on this board from a row in the middle of the scoring table. Extras are reported separately and never folded into the score, but a model that flags this many unplanted defects while fixing fewer planted ones is spending its attention somewhere the planted total cannot see. *** IT KILLED AN ALL-TIME SURVIVOR *** the first run fixed a planted bug that no scored pass by any model had ever fixed before - a settings-persistence ordering defect where the stored value is written before the live switch it is supposed to describe has landed. That is a first for this row and it stands regardless of where the mean settles. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows; the choice was a one-version harness delta or no row at all. Read the comparison above with that in it. (The rejection message is generic - a deliberately fake slug returns it verbatim - so it is not evidence about any model's existence.) *** WALL AND COST *** Walls 60.4, 58.2 minutes, mean 59.3; cost $10.03, $8.17, mean $9.10. COST IS A LIST ESTIMATE AND A FLOOR - token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Legs were judged by grok-4.7 with a single exception - repo 1 of the second run drew grok-4.5 - blinded to the arm's identity. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The runs shared the laptop with legs from other vendors, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 6, 13, 16, 20, 21, 22, 25, 27, 28, 29, 30, 33, 34, 37, 39, 40, 43, 44, 45, 47, 49, 50, 61, 62, 71, 76, 77, 82, 83, 84, 98, 99, 100, 102 ], "bug_hits": { "6": 1, "13": 2, "16": 2, "20": 2, "21": 2, "22": 1, "25": 2, "27": 1, "28": 1, "29": 2, "30": 2, "33": 2, "34": 2, "37": 2, "39": 2, "40": 2, "43": 2, "44": 2, "45": 2, "47": 2, "49": 2, "50": 2, "61": 1, "62": 2, "71": 2, "76": 2, "77": 2, "82": 1, "83": 2, "84": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 353, "turns_repo1": 156, "turns_repo2": 196, "group_rank": 200 }, { "id": "GPT-5.6 Sol", "n_runs": 1, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-07-26", "color": "#5AB99C", "fixed": 31, "repo1_fixed": 13, "repo2_fixed": 18, "partial": 4, "claimed_only": 1, "extras": 38, "false_positive_fixes": 0, "wall_min": 70.2, "cost_usd": 29.16, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded by the Jul 31 re-run on the same setting (serving-side change, not a new model).", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 25, 29, 30, 33, 34, 39, 40, 43, 44, 45, 46, 47, 49, 50, 62, 71, 76, 77, 80, 82, 83, 84, 92, 95, 97, 98, 99, 102 ], "turns": 325, "turns_repo1": 122, "turns_repo2": 203, "group_rank": 402 }, { "id": "Union Alpha (stealth; free) - mean of 3", "n_runs": 3, "model": "Pareto (ex-Union Alpha)", "vendor": "Unbiased", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-17", "color": "#EC13EC", "fixed": 30.7, "repo1_fixed": 16, "repo2_fixed": 14.7, "partial": 1.3, "claimed_only": 2, "extras": 10.3, "false_positive_fixes": 0, "wall_min": 38.2, "cost_usd": 4.81, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "Runs scored 32, 34 and 26 of 105 (repo 1: 16, 19, 13 - repo 2: 16, 15, 13). Mean 30.7, range 8. *** THE SPREAD IS THE FINDING, AND IT IS WIDE. *** Three runs of one fixed ARM DEFINITION on what was then a free stealth slug, revealed 2026-09-17 as unbiased/pareto - a COMPOSITE that runs several models in parallel per request and selects or synthesises the output. Read every number here as a blend's, not a model's. The published single run sat in the middle of a range that no single run could have shown. Read the mean as an estimate with a known and fairly wide spread, not as a score. *** ONE MEMBER RAN ITSELF INTO THE CONTEXT CEILING AND THE OTHER TWO DID NOT. *** r3's repo-1 leg peaked at 197,452 tokens against this endpoint's 262,144 ceiling and was compacted once by the harness; r1 peaked at 23,028 and neither r1 nor r2 compacted at all. Every configuration parameter was held fixed - same slug, same single endpoint, same harness build 2.1.274, same prompt, same repos, same judge - so what differed is what the MODEL did: r3 made about 2.3x the tokens and 1.9x the wall of r1. It is averaged in rather than dropped, because dropping a member for behaving differently selects runs on their behaviour. The automated regime check returns CANNOT TELL on this arm (r2's transcript reports no context accounting), so this is stated from the gateway log rather than implied by a check that did not run. The replication is therefore UNVERIFIED rather than verified: beyond identical arm definitions and the gateway log, nothing confirms the three runs shared a regime. *** WHAT THESE RUNS MEASURED WAS ZERO, AND ONE FIGURE BEHIND THAT WAS REPAIRED. *** The slug was priced at $0/$0 in preview and billed nothing: a live call through this arm's own pinned path returned usage.cost 0, upstream_inference_cost 0, and a credits delta of exactly $0.000000. r3's two legs nonetheless recorded 0.0787 and 0.0662, because for aggregator arms the runner measures a CREDITS DELTA across the leg - so any other consumer on the same account lands in it, and another session was fingerprinting this model against PAID reference models through r3's window. Both cells were repaired to 0 and both originals are preserved in the metrics receipts. r1 and r2 measured 0 unaided. WHAT THESE RUNS WOULD COST AT THE REVEALED PRICE: $14.43 for all three, $4.81 per run, repricing their own measured token counts at unbiased/pareto list ($2.50 prompt, $7.50 completion, $0.25 cached read per million). So the cell reads $4.81 and is marked cost_list, which the glossary defines as a token-count estimate at published list rates and not a bill. It was $0.00 until the preview ended; an honest zero then stopped supporting the only comparison the column exists for, so the figure was repriced generator-side. Nothing was actually spent, and the published receipts still say so. *** THE SLUG SERVED MORE THAN ONE VOCABULARY. *** Predicted before the reveal and confirmed against the revealed endpoint: at least three backends answer to this slug. The benchmark runs themselves do NOT look mixed: the three replicates agree at mean Jaccard 0.628 on their fixed-bug sets against a board median of 0.632 over 24 replicate families, which is ordinary. The probe evidence suggests the router was length-sensitive, so short probes and long agentic runs need not have landed on the same backend. The score is what was measured; that it describes a single model is not something this board can claim. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT. *** A model may not judge itself or a sibling, and at run time the lab was unknown, so the scoring pass ran on grok-4.6 rather than the board default. The reveal makes that the right call for a firmer reason than the one recorded: a COMPOSITE has no single vendor to check a judge against, and third-party analysis of the panel places GPT-6 Astra on the escalation path - the default judge's own family, inside the thing being judged. All three runs were scored by grok-4.6. r1 was ALSO judged by codex into a separate scratch pass, and the two judges selected the SAME 32 bugs - identical sets on both repos, not merely equal counts. *** NO EFFORT DIAL EXISTS ON THIS ROUTE: supported_parameters carries no reasoning field, the model returns reasoning_tokens=0, and an invented tier is refused by the gateway in 0.2s without reaching the provider. So effort is 'default' and deliberately not 'inert_default' - nothing was asserted, and there was no dial to probe.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 29, 30, 33, 34, 35, 36, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 68, 70, 74, 76, 77, 78, 79, 80, 82, 83, 84, 91, 92, 96, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 2, "23": 1, "25": 3, "29": 3, "30": 3, "33": 3, "34": 3, "35": 1, "36": 2, "37": 2, "39": 3, "40": 3, "41": 3, "43": 2, "44": 3, "45": 1, "47": 3, "49": 3, "50": 3, "68": 1, "70": 2, "74": 1, "76": 2, "77": 3, "78": 2, "79": 1, "80": 2, "82": 2, "83": 3, "84": 3, "91": 1, "92": 2, "96": 1, "98": 1, "99": 2, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 113, "turns_repo1": 67, "turns_repo2": 46, "group_rank": 6 }, { "id": "Opus 5.5 (medium effort) - mean of 3", "n_runs": 3, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 30.3, "repo1_fixed": 12.7, "repo2_fixed": 17.7, "partial": 2.3, "claimed_only": 1, "extras": 13.3, "false_positive_fixes": 0, "wall_min": 17.0, "cost_usd": 15.68, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 30, 30 and 31 of 105 - spread 1. Read the mean with that width, not as a point. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at medium effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 30, 30 and 31 of 105 (repo 1: 11, 14, 13 of 45 - repo 2: 19, 16, 18 of 60). Mean 30.3, spread 1. *** THIS SUPERSEDES OPUS 5.5 (MEDIUM EFFORT) - MEAN OF 2 (30.0 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 17.2, 17.5, 16.4 minutes, mean 17.0; cost $16.55, $14.80, $15.71, mean $15.69. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 15, 13 and 12 across the runs, mean 13.3 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 25, 26, 28, 29, 30, 33, 34, 36, 39, 43, 45, 46, 47, 49, 50, 60, 62, 70, 71, 76, 77, 80, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 3, "20": 3, "22": 3, "25": 2, "26": 2, "28": 1, "29": 3, "30": 3, "33": 3, "34": 3, "36": 1, "39": 3, "43": 3, "45": 3, "46": 3, "47": 3, "49": 3, "50": 2, "60": 3, "62": 2, "70": 1, "71": 3, "76": 3, "77": 3, "80": 1, "82": 2, "83": 3, "84": 3, "92": 2, "95": 1, "97": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 189, "turns_repo1": 66, "turns_repo2": 123, "group_rank": 203 }, { "id": "Opus 5.5 (medium effort) - mean of 2", "n_runs": 2, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 30, "repo1_fixed": 12.5, "repo2_fixed": 17.5, "partial": 3, "claimed_only": 0.5, "extras": 14, "false_positive_fixes": 0, "wall_min": 17.3, "cost_usd": 15.67, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (30.3 of 105). This row reported 30.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 30 to 31. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 30 and 30 of 105 - spread 0. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at medium effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 30 and 30 of 105 (repo 1: 11, 14 of 45 - repo 2: 19, 16 of 60). Mean 30.0, spread 0. *** THIS SUPERSEDES OPUS 5.5 (MEDIUM EFFORT) (30 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 17.2, 17.5 minutes, mean 17.4; cost $16.55, $14.80, mean $15.68. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 15 and 13 across the runs, mean 14.0 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 25, 26, 28, 29, 30, 33, 34, 36, 39, 43, 45, 46, 47, 49, 50, 60, 62, 70, 71, 76, 77, 82, 83, 84, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 2, "20": 2, "22": 2, "25": 1, "26": 1, "28": 1, "29": 2, "30": 2, "33": 2, "34": 2, "36": 1, "39": 2, "43": 2, "45": 2, "46": 2, "47": 2, "49": 2, "50": 1, "60": 2, "62": 2, "70": 1, "71": 2, "76": 2, "77": 2, "82": 1, "83": 2, "84": 2, "92": 1, "95": 1, "97": 2, "98": 2, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 190, "turns_repo1": 70, "turns_repo2": 120, "group_rank": 203 }, { "id": "Opus 5.5 (medium effort)", "n_runs": 1, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-22", "color": "#DE6C48", "fixed": 30, "repo1_fixed": 11, "repo2_fixed": 19, "partial": 3, "claimed_only": 1, "extras": 15, "false_positive_fixes": 0, "wall_min": 17.2, "cost_usd": 16.55, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (30.0 of 105). This row reported 30 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 30 to 30. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean. This is the medium rung of a four-rung same-day sweep (medium 30, high 32, xhigh 35, max 43 of 105); the ladder is monotone, but no gap in it exceeds the 13-point run-to-run spread this board has measured on a single identical configuration, so read the direction, not the ordering of adjacent rungs. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. The wall figure is contended.", "note": "Anthropic's Opus 5.5 at medium effort through Claude Code on the first-party API. 30 of 105: 11 of 45 on repo 1 and 19 of 60 on repo 2, 15 genuine extras, 17.2 minutes, $16.55. ONE RUN. *** THIS IS ONE RUNG OF A FOUR-RUNG SWEEP *** All four Opus 5.5 rows on this board are the same arm definition with one character changed - the effort dial - run on the same day, on the same machine, against the same two repos, judged the same way. The ladder is monotone: medium 30, high 32, xhigh 35, max 43 of 105, costing $16.55, $23.03, $35.14 and $60.49. *** BUT READ THE LADDER AS A TREND, NOT AS FOUR LEVELS *** every rung is a single run, and this board has already measured a 13-POINT spread between two runs of one identical max-effort configuration elsewhere. No gap on this ladder is that wide - the largest, max over xhigh, is 8. So the direction is credible across four points; the ordering of any two adjacent rungs is not resolved at n=1, and nothing here says medium is reliably worse than the rung above it. *** THE DIAL BINDS - MEASURED, NOT ASSUMED *** this board has caught several third-party routes that accept a reasoning field and ignore it, so 'first party' is not taken on trust. Token spend on repo 2 rises monotonically with the dial across the sweep - 33.4M at medium, 46.8M at high, 82.7M at xhigh, 150.2M at max, a 4.5x span - and wall rises with it. This rung burned 33.4M. Whatever else is true, the setting is reaching the model and changing how hard it works. *** THE CONTEXT WINDOW, WHICH IS WHY THESE ROWS DO NOT COMPARE TO THE OPUS 5 LADDER *** this arm runs the model string claude-opus-5-5[1m] and therefore a 1M window: it held a peak of 407,965 tokens on repo 2 and crossed ZERO compaction boundaries, carrying the repo whole through every turn. Every Opus 5 row on this board runs a bare slug, which the CLI treats as 200K, and those runs compacted repeatedly mid-task. Comparing this ladder to the Opus 5 ladder moves two variables at once - model version AND how much of the codebase the model could hold while reasoning about it. The [1m] suffix was verified on CLI 2.1.280 rather than assumed (an earlier arm on this board requested a 1M window and was silently normalised to the bare slug on 2.1.272); init.model returns 'claude-opus-5-5[1m]' verbatim. Autocompact ran at the CLI's own default - a user-level CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=30 that had been reaching legs on this machine was removed before the sweep and its absence re-verified in a leg-shaped probe. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Opus 5.5 rates: $4 input, $20 output, $0.20 cache read - a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x - and $8 cache write, the 1-hour rate, which is what this harness actually writes. Not an invoice, hence `list`. *** RECEIPTS *** Both legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The run shared the laptop with benchmark legs from another vendor, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 16, 20, 22, 28, 29, 30, 33, 34, 39, 43, 45, 46, 47, 49, 50, 60, 62, 70, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 197, "turns_repo1": 74, "turns_repo2": 123, "group_rank": 203 }, { "id": "Grok 4.7 (medium)", "n_runs": 1, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "medium", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 30, "repo1_fixed": 11, "repo2_fixed": 19, "partial": 3, "claimed_only": 0, "extras": 30, "false_positive_fixes": 0, "wall_min": 32.4, "cost_usd": 25.15, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-21 by the n=3 row at the same configuration (26.7 of 105). The three runs scored 30, 23 and 27, so this single draw at 30 sat at the edge of a 7-point spread it could not show. Nothing it claimed is retracted; it was published as an n=1 row and read as one. Kept rather than deleted because it is the receipt for what the board showed for part of 2026-09-21.", "caveat": "ONE RUN, not a mean. The same model at its top rung spanned 25 to 31 of 105 across three runs, so read this number with several points of slack in either direction. Cost is a reconstructed FLOOR rather than a bill, and it uses a per-session arithmetic the older grok rows on this board do not yet use; the note gives both figures.", "note": "Grok 4.7 at medium through xAI's own CLI - the MEDIUM rung, one step below the CLI default. 30 of 105: 11 of 45 on repo 1 and 19 of 60 on repo 2, 30 genuine extras, 32.4 minutes, a reconstructed floor of $25.15. ONE RUN. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 (medium) scored 23.0 of 105 (7 on repo 1, 16 on repo 2), also a single run, for a difference of +7. Both sides are n=1, so this is one draw against one draw. The scale of the doubt is not a guess: three runs of the 4.6 configuration one rung up spanned 27 to 30 of 105, and three runs of this model at that rung spanned 25 to 31 - so a gap of a few points between two single runs is inside what the same configuration produces run to run. *** WHERE IT SITS ON ITS OWN DIAL *** This model at xhigh is 28.8 of 105, published as a mean of four. This rung is 30, a single run. The 4.6 dial for comparison ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, which is the most interesting thing on it, and the low rung is where that dial had real room. Whether either shape repeats here is a question the rungs answer together: read this row beside its siblings rather than on its own. *** THE TIER WAS CHECKED, NOT ASSUMED, AND FOR THE LOWER RUNGS THAT MATTERS MORE *** A zero-token ACP readback before launch returned reasoning_effort medium for a requested medium. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except this readback - the run would simply be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. The served model is named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. *** COST IS A FLOOR *** The grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates, the same 2/0.5/6 per MTok basis. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic the other grok rows still use it would read $28.86. It is a floor twice over: xAI bills a request whose prompt reaches 200K at DOUBLE those rates. *** RECEIPTS *** Both legs exited 0. Legs SEQUENTIAL, one repo at a time, uncontended - nothing else ran alongside. The agent spawned 3 session(s) on repo 1 and 5 on repo 2 (133 and 199 inference requests); that count varies run to run on identical arm definitions for this model, and it tracks cost far more closely than score. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 37, 39, 43, 45, 47, 49, 50, 56, 57, 60, 62, 71, 76, 78, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 457, "turns_repo1": 165, "turns_repo2": 292, "group_rank": 3 }, { "id": "GPT-6 Sol (max effort) - mean of 3", "n_runs": 3, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 29.3, "repo1_fixed": 15.3, "repo2_fixed": 14, "partial": 2, "claimed_only": 0.3, "extras": 41.3, "false_positive_fixes": 0, "wall_min": 63.4, "cost_usd": 9.33, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 32, 30 and 26 of 105 - spread 6. Read the mean with that width, not as a point. It also runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; the harness is therefore not held fixed across that comparison. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. Wall figures are contended.", "note": "OpenAI's GPT-6 Sol at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. Published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 32, 30 and 26 of 105 (repo 1: 18, 16, 12 of 45 - repo 2: 14, 14, 14 of 60). Mean 29.3, spread 6. *** THIS SUPERSEDES GPT-6 SOL (MAX EFFORT) - MEAN OF 2 (31.0 OF 105) *** *** THE NEWER CHECKPOINT STILL DOES NOT LEAD ITS OWN FAMILY *** GPT-6 Astra publishes 45 (mean of 3) of 105 at max and 43 at xhigh on this same benchmark; GPT-5.6 Sol publishes 43.5 (mean of 2) at max. This row lands at 29.3, below all three, at a fraction of the cost. The single run that first showed this was one draw on a board that has measured a 13-point spread between two runs of one identical max-effort configuration - which is exactly why it was replicated rather than left to stand. *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string gpt-6-sol, same effort, same timeout, same pinned executable, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** IT FINDS A LOT OF THINGS NOBODY PLANTED *** 48, 34 and 42 genuine extras across the runs, mean 41.3 - real defects in the repos that are not part of the planted set, among the highest counts on this board from a row in the middle of the scoring table. Extras are reported separately and never folded into the score, but a model that flags this many unplanted defects while fixing fewer planted ones is spending its attention somewhere the planted total cannot see. *** IT KILLED AN ALL-TIME SURVIVOR *** the first run fixed a planted bug that no scored pass by any model had ever fixed before - a settings-persistence ordering defect where the stored value is written before the live switch it is supposed to describe has landed. That is a first for this row and it stands regardless of where the mean settles. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows; the choice was a one-version harness delta or no row at all. Read the comparison above with that in it. (The rejection message is generic - a deliberately fake slug returns it verbatim - so it is not evidence about any model's existence.) *** WALL AND COST *** Walls 60.4, 58.2, 71.6 minutes, mean 63.4; cost $10.03, $8.17, $9.81, mean $9.34. COST IS A LIST ESTIMATE AND A FLOOR - token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Legs were judged by grok-4.7 with a single exception - repo 1 of the second run drew grok-4.5 - blinded to the arm's identity. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The runs shared the laptop with legs from other vendors, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 6, 13, 16, 20, 21, 22, 25, 27, 28, 29, 30, 33, 34, 37, 39, 40, 43, 44, 45, 47, 49, 50, 61, 62, 71, 76, 77, 82, 83, 84, 98, 99, 100, 102 ], "bug_hits": { "6": 1, "13": 3, "16": 3, "20": 3, "21": 2, "22": 1, "25": 3, "27": 2, "28": 1, "29": 3, "30": 3, "33": 3, "34": 3, "37": 2, "39": 3, "40": 3, "43": 2, "44": 2, "45": 3, "47": 3, "49": 3, "50": 3, "61": 2, "62": 3, "71": 3, "76": 3, "77": 3, "82": 1, "83": 3, "84": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": 352, "turns_repo1": 163, "turns_repo2": 189, "group_rank": 200 }, { "id": "GPT-6.1 Sol (medium effort) - mean of 2", "n_runs": 2, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-30", "color": "#2BA580", "fixed": 29, "repo1_fixed": 16, "repo2_fixed": 13, "partial": 3, "claimed_only": 0, "extras": 30.5, "false_positive_fixes": 0, "wall_min": 29.8, "cost_usd": 1.76, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Two runs of an identical configuration scoring 28 and 30 of 105 - spread 2. Read the mean with that width - two runs is a thin mean. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates. Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at medium effort through OpenAI's own Codex CLI, on a ChatGPT account, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 28 and 30 of 105 (repo 1: 16, 16 - repo 2: 12, 14). Mean 29.0, range 2. *** THIS SUPERSEDES GPT-6.1 SOL (MEDIUM EFFORT) (28 OF 105) *** *** THE REPLICATE IS BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, pinned CLI, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. *** WALL, COST AND TURNS *** walls 34.0, 25.6 minutes; cost $1.85, $1.67; turns 95, 89. *** GENUINE EXTRAS *** 32, 29 across the runs, never folded into the score. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at low 31, GPT-6.1 Sol at high 36.5 (mean of 2), GPT-6.1 Sol at xhigh 42.7 (mean of 3), GPT-6.1 Sol at max 44.3 (mean of 3). At the same tier the board publishes GPT-6 Sol at medium 14, GPT-6 Astra at medium 34, Sonnet 5.5 at medium 23.3 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at medium. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 35, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 62, 68, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "22": 2, "25": 2, "29": 2, "30": 2, "33": 2, "34": 2, "35": 1, "37": 1, "39": 2, "40": 2, "41": 2, "43": 2, "44": 2, "45": 2, "47": 2, "49": 2, "50": 2, "62": 1, "68": 1, "76": 2, "77": 1, "82": 2, "83": 2, "84": 2, "92": 1, "97": 1, "98": 2, "99": 2, "100": 2, "102": 1 }, "coverage_runs": 2, "turns": 92, "turns_repo1": 55, "turns_repo2": 37, "group_rank": 103 }, { "id": "GPT-5.6 Sol (medium effort)", "n_runs": 1, "model": "GPT-5.6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-06", "color": "#5AB99C", "fixed": 29, "repo1_fixed": 13, "repo2_fixed": 16, "partial": 1, "claimed_only": 0, "extras": 24, "false_positive_fixes": 0, "wall_min": 47.6, "cost_usd": 15.77, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": "The Sol dial: medium 29 -> high 34 -> xhigh 39 -> max 42.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 39, 41, 44, 45, 46, 47, 49, 50, 59, 62, 76, 78, 80, 83, 84, 92, 97, 98, 99, 102 ], "turns": 143, "turns_repo1": 67, "turns_repo2": 76, "group_rank": 403 }, { "id": "Grok 4.7 (xhigh) - mean of 5", "n_runs": 5, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified", "date": "2026-10-08", "color": "#414852", "fixed": 29, "repo1_fixed": 13, "repo2_fixed": 16, "partial": 1.4, "claimed_only": 0.2, "extras": 32.2, "false_positive_fixes": 0, "wall_min": 44.7, "cost_usd": 22.96, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "A mean of five runs (25 to 31 of 105) against a Grok 4.6 row that is a mean of three. The fifth run came 17 days after the first four, on the same CLI build with the window held at the same 500K, and took its two repos at the same time where runs 1-4 took them one after the other, so its wall is contended. Cost is a reconstructed floor at API list rates.", "note": "Grok 4.7 at xhigh through xAI's own CLI, published as the MEAN OF FIVE INDEPENDENT RUNS of one configuration. Runs scored 25, 30, 31, 29 and 30 of 105 (repo 1: 12, 12, 16, 12, 13 - repo 2: 13, 18, 15, 17, 17). Mean 29.0, range 6. *** THE FIFTH RUN CAME 17 DAYS LATER AND CHANGED NOTHING *** Runs 1-4 ran on 2026-09-21; run 5 on 2026-10-08, to test whether the model behind the grok-4.7 slug had improved since release. It scored 30, inside the first four's range, and moved the mean from 28.8 to 29.0. A clean run of an identical configuration is added to its row whatever it scores. Grok 4.6 at the same tier, a mean of three, stands at 28.7 of 105: the tie between the two generations holds at n=5. *** HOW RUN 5 WAS HELD TO THE SAME CONFIGURATION *** Same model, effort, prompt and judge, and the same CLI build as the first four (grok 1.0.30), run from a pinned copy because the installed CLI had auto-updated. The model was pinned per session and confirmed by the session itself. The context window needed restoring: by 2026-10-08 xAI's model metadata gave grok-4.7 a 256K default (500K optional), and the CLI reads it from there, so the run set 500K, the window runs 1-4 had and the size their prompts reached (~285K at peak). A zero-token readback before launch matched the 09-21 gate: grok-4.7, reasoning effort xhigh, 500,000-token window. *** WHAT DIFFERS, AND WHAT IT DOES NOT TOUCH *** Runs 1-4 took their two repos one after the other; run 5 took them at the same time, which is why this row's name drops 'seq'. That can only inflate run 5's wall, and run 5 was still the fastest of the five. Scores, tokens and cost are unaffected. Run 5's repo-2 leg was judged twice (a duplicate pass): 16 and 17 strict on the same diff. The board counts the newer pass, 17; the other would put the mean at 28.8. *** BUGS NOBODY PLANTED *** Genuine unplanted extras ran 26, 35, 34, 32 and 34 (mean 32.2). Reported separately and never added to the score. Claimed-only fixes: 0, 1, 0, 0 and 0. *** COST IS A RECONSTRUCTED FLOOR *** $16.55, $23.04, $29.93, $22.01 and $23.26 per run, mean $22.96, all on the same per-session arithmetic. The grok CLI reports context occupancy, not cumulative usage, so the cost is rebuilt from the CLI's own session logs at xAI's API list rates ($2 / $0.50 / $6 per MTok). It is a floor: checked on run 5 against exact token counts, it sat about 10% under list before xAI's double rate on prompts of 200K and up, which these runs reach and this method cannot see. *** THE TIER IS THE MODEL CEILING *** A zero-token readback before every launch returned reasoning effort xhigh; grok-4.7 advertises low, medium, high and xhigh, so nothing above xhigh exists to ask for. *** RECEIPTS *** All ten legs exited 0. Per-run wall 39.6, 47.3, 51.8, 45.4, 39.4 minutes. 6 of the 10 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 33, 34, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 49, 50, 56, 57, 60, 62, 70, 71, 76, 82, 83, 84, 87, 89, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 4, "16": 5, "20": 5, "21": 1, "22": 3, "23": 2, "25": 5, "27": 1, "28": 1, "29": 4, "30": 5, "33": 2, "34": 3, "37": 5, "38": 1, "39": 4, "40": 2, "41": 1, "43": 5, "44": 1, "45": 5, "46": 1, "47": 5, "49": 4, "50": 4, "56": 1, "57": 1, "60": 5, "62": 5, "70": 1, "71": 5, "76": 5, "82": 3, "83": 5, "84": 4, "87": 2, "89": 1, "91": 2, "92": 5, "95": 3, "97": 1, "98": 3, "99": 5, "100": 5, "102": 4 }, "coverage_runs": 5, "turns": 503, "turns_repo1": 173, "turns_repo2": 329, "group_rank": 1 }, { "id": "Fable 5.1 (low effort)", "n_runs": 1, "model": "Fable 5.1", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-02", "color": "#D9542B", "fixed": 29, "repo1_fixed": 13, "repo2_fixed": 16, "partial": 1, "claimed_only": 1, "extras": 6, "false_positive_fixes": 0, "wall_min": 33.2, "cost_usd": 33.0, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Mixed judges within this row: repo 1 by GPT-5.5, repo 2 by Grok 4.5 after the Codex judge returned a malformed verdict on that packet. Every other Fable 5.1 row is Codex-judged throughout. The repo 2 leg was also re-run: the first attempt spanned a network outage that voided its wall clock (862 vs 2183 output tokens/min) and scored 18/60 under Codex, against 16/60 under Grok here. The diff and the judge both changed between those two, so the difference cannot be attributed to either one.", "note": null, "fixed_bugs": [ 13, 16, 20, 22, 28, 29, 30, 33, 34, 37, 39, 41, 43, 47, 49, 50, 60, 62, 71, 76, 77, 82, 83, 84, 97, 98, 99, 100, 102 ], "turns": 234, "turns_repo1": 95, "turns_repo2": 139, "group_rank": 4 }, { "id": "Fable 5.1 (xhigh effort)", "n_runs": 1, "model": "Fable 5.1", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-10", "color": "#D9542B", "fixed": 29, "repo1_fixed": 13, "repo2_fixed": 16, "partial": 1, "claimed_only": 5, "extras": 4, "false_positive_fixes": 0, "wall_min": 59.5, "cost_usd": 53.61, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "One run per tier, five tiers - so this is five samples describing a surface, not a measured response curve. The board's own variance evidence (three runs of one model at one setting scoring 16, 13 and 17) means a twelve-point spread across four single runs is entirely consistent with noise, which is precisely the point: if noise can produce this ordering, the ordering carries no information. The one claim that survives is max against the rest, and even that rests on a single run at each end.", "note": "The row that completes Fable 5.1's dial, and the one that shows the dial does not behave. All five tiers now exist at one run each: low 29, medium 21, high 33, xhigh 29, max 43. The four below max span twelve points in no order whatsoever - xhigh lands BELOW high and exactly level with low, medium is the worst of the five - while max sits ten clear above the best of them. The run itself is clean: both legs exited success with a report written, 13/45 on repo 1 and 16/60 on repo 2. So the honest reading of this model's effort control is that only the top of it is doing visible work on this benchmark; between low and xhigh the label does not predict the score. That matters because the effort column is the thing readers most often take as an ordering, and here it is not one.", "fixed_bugs": [ 13, 16, 20, 21, 22, 27, 29, 30, 33, 34, 39, 43, 45, 47, 49, 50, 62, 70, 71, 76, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 420, "turns_repo1": 208, "turns_repo2": 212, "group_rank": 1 }, { "id": "Fable 5 (max effort)", "n_runs": 1, "model": "Fable 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-08-01", "color": "#DC603A", "fixed": 29, "repo1_fixed": 12, "repo2_fixed": 17, "partial": 0, "claimed_only": 0, "extras": 5, "false_positive_fixes": 0, "wall_min": 57.3, "cost_usd": 112.4, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 22, 25, 28, 29, 30, 39, 41, 43, 44, 45, 47, 49, 50, 60, 62, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 315, "turns_repo1": 104, "turns_repo2": 211, "group_rank": 100 }, { "id": "Grok 4.7 (xhigh) seq - mean of 4", "n_runs": 4, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 28.8, "repo1_fixed": 13, "repo2_fixed": 15.8, "partial": 1.3, "claimed_only": 0.2, "extras": 31.8, "false_positive_fixes": 0, "wall_min": 46.0, "cost_usd": 22.89, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-10-09 by the mean of five (29.0 of 105). A fifth run of the same configuration, made 17 days later, scored 30 and moved the mean by +0.2. Nothing this row claimed is retracted. Kept as the receipt for what the board showed from 2026-09-21 to 2026-10-09.", "caveat": "A mean of FOUR runs against a 4.6 row that is a mean of three - the samples are not the same size. The four span 25 to 31 of 105 against 4.6's 27 to 30, so the tie still rests on the wider distribution. Cost uses a per-session reconstruction the other grok rows do not yet use; the note gives both figures and the gap holds either way. Two of the four runs share a machine with each other, so only the wall figures from runs 1 and 4 support a speed comparison. The runs spawned different numbers of subagent sessions on identical arm definitions, which the board's regime check cannot verify on this harness.", "note": "Grok 4.7 at xhigh through xAI's own CLI, published as the MEAN OF FOUR INDEPENDENT RUNS. Runs scored 25, 30, 31 and 29 of 105 (repo 1: 12, 12, 16, 12 - repo 2: 13, 18, 15, 17). Mean 28.8, range 6. *** THE HEADLINE IS STILL THAT NOTHING MOVED *** Grok 4.6 at the SAME tier, a mean of three, stands at 28.7 of 105. This row is 28.8. The fourth run was added specifically to test whether the tie was an artefact of three draws; it was not - the headline moved by +0.1 of a bug. This supersedes the mean-of-3 row published earlier the same day, which said the same thing on less evidence. A tie is a measurement, and publishing only when a number improves would make this a scoreboard for vendors rather than a benchmark. *** THE IDENTICAL TOTAL HIDES A SWAP, AND ONE REPO IS NEARLY CONSTANT *** Repo 1 scored 12, 12, 16, 12 of 45 against 4.6's mean of 11.7; repo 2 scored 13, 18, 15, 17 of 60 against 17.0. Three of the four repo-1 runs landed on the same number, which makes the fourth look like the outlier rather than the ceiling; repo 2 scatters across five points. Both movements sit inside the spread of the runs producing them. *** THE SPEED QUESTION IS SETTLED, AND THE ANSWER IS NO *** Runs 2 and 3 overlapped each other on one machine, so their walls are biased upward and the earlier row published this comparison as undecidable. Runs 1 and 4 each had the machine to themselves: 39.6 and 45.4 minutes, a mean of 42.5 against 43.1 for the three 4.6 runs. That is a difference of under two minutes on runs that take three quarters of an hour. Any earlier reading that this generation is faster came from a single run and does not survive a second clean one. *** WHAT DOES MOVE, AND IT MOVES THE WRONG WAY: COST *** Reconstructed floors of $16.54, $23.04, $29.94, $22.01 for a mean of $22.88, against $16.89 for the three 4.6 runs on the SAME arithmetic - about 35% more for the same score. READ THE COST COLUMN CAREFULLY: the reconstruction used to take the largest context in a run and charge every request in that run against it, which overstates any run that fans out into subagents by up to 25%, and it now measures each session separately. This row carries the corrected figure; the 4.6 row still shows $18.60 on the old arithmetic and has not yet been restated. On that same old arithmetic these four runs come to $27.10. The gap is about a third either way - what is inconsistent across the board right now is the arithmetic, not the finding. Both are floors twice over: the grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates on the same 2/0.5/6 per MTok basis, and xAI bills a request whose prompt reaches 200K at DOUBLE those rates, which these runs did, by an amount this method cannot see. *** AND WHAT MOVES IN 4.7'S FAVOUR: BUGS NOBODY PLANTED *** Genuine extras - real defects in the repo that are not part of the planted set - ran 26, 35, 34 and 32 across the four runs, a mean of 31.8, against 18.0 for 4.6: roughly 76% more, and the most stable difference between the two models in this whole wave. The most defensible reading is that 4.7 finds materially more real bugs than 4.6 and no more of the ones being counted. Whether that is a better bug-hunter or a worse instruction-follower is a judgment the number does not settle. *** PER BUG, THE TWO MODELS ARE NEARLY THE SAME MODEL *** Measured across the first three runs a side: on repo 1 each touched 18 distinct planted bugs with 15 shared; on repo 2, 21 against 22 with 18 shared. There is not one bug on either repo that 4.6 found on all three runs and 4.7 never found, and exactly one the other way. Every other bug unique to one model was a single-run hit - noise, not capability. *** THE RUNS DID NOT BEHAVE ALIKE, AND THE AUTOMATIC CHECK CANNOT SEE IT HERE *** The regime check that refuses a mean over mismatched configurations parses Claude Code transcripts, so it returns NOT CHECKED for every grok row on this board. The CLI's own session dirs show what it would look for: sessions spawned per leg (repo 1 / repo 2) ran run 1 1 and 1; run 2 1 and 3; run 3 5 and 1; run 4 2 and 5. Grok 4.6 never once ran a leg as a single session across its six; this model did it four times out of eight, and when it does fan out it commits far harder - up to 206 inference requests against 4.6's 88 to 128. Across these four runs fan-out tracks COST almost mechanically, and what it does to SCORE is unresolved rather than nothing: ordered by inference requests the four runs scored 25, 30, 29 and 31, so the heaviest fan-out landed mid-pack but the run with NO fan-out landed LAST, not mid-pack. The rank correlation against score is positive and four runs cannot separate that from chance. CORRECTED 2026-09-21: an earlier version of this note said the no-fan-out run landed mid-pack and that fan-out tracked score not at all; the first half was simply wrong - that run scored lowest of the four - and the second was stated more firmly than four runs support. The arm definitions are byte-identical clones, so this is emergent agent behaviour rather than configuration. An OBSERVATION over four runs, not a mechanism. *** THE TIER IS THE MODEL CEILING AND THAT WAS CHECKED, NOT ASSUMED *** A zero-token ACP readback before each launch returned reasoning_effort xhigh for a requested xhigh, and the enum grok-4.7 advertises is low, medium, high, xhigh with high as the CLI default - nothing above xhigh exists to ask for. The grok CLI silently clamps an unknown effort string, so an echo is the only thing that makes this label honest. The served model is named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. *** RECEIPTS *** All eight legs exited 0. Per-run wall 39.6, 47.3, 51.8, 45.4 minutes; runs 1 and 4 uncontended, runs 2 and 3 overlapping each other. Legs SEQUENTIAL within every run, one repo at a time - that is what seq means here. Judge: blind Codex gpt-5.5 on all eight legs, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 27, 28, 29, 30, 33, 34, 37, 38, 39, 40, 43, 45, 46, 47, 49, 50, 57, 60, 62, 70, 71, 76, 82, 83, 84, 87, 89, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 4, "20": 4, "22": 2, "23": 1, "25": 4, "27": 1, "28": 1, "29": 4, "30": 4, "33": 2, "34": 3, "37": 4, "38": 1, "39": 4, "40": 2, "43": 4, "45": 4, "46": 1, "47": 4, "49": 3, "50": 3, "57": 1, "60": 4, "62": 4, "70": 1, "71": 4, "76": 4, "82": 2, "83": 4, "84": 3, "87": 1, "89": 1, "91": 2, "92": 4, "95": 3, "97": 1, "98": 2, "99": 4, "100": 4, "102": 3 }, "coverage_runs": 4, "turns": 500, "turns_repo1": 176, "turns_repo2": 324, "group_rank": 1 }, { "id": "Grok 4.6 (xhigh) seq - mean of 3", "n_runs": 3, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified", "date": "2026-09-14", "color": "#757A82", "fixed": 28.7, "repo1_fixed": 11.7, "repo2_fixed": 17, "partial": 2.7, "claimed_only": 0.7, "extras": 18, "false_positive_fixes": 0, "wall_min": 43.1, "cost_usd": 18.6, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 27 to 30. This row SUPERSEDES the single-run sequential row (27/105) published 2026-08-28, which was the LOWEST of the three. Every figure on this row is a MEAN of the three runs, carried to one decimal - 28.7 of 105, not a 29 that happens to be an average. Cost is a reconstructed FLOOR rather than a bill, and the three floors span 16.96 to 20.88 dollars, so the cost column here is a mean of three estimates and not a price.", "note": "The FIRST configuration outside Meta's effort dial to be run three times, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 27, 30 and 29 of 105 (repo 1: 8, 14, 13 - repo 2: 19, 16, 16). Mean 28.7, range 3. *** WHAT THIS CORRECTS IS AN INFERENCE, NOT A NUMBER *** The single run this replaces scored 27, and its note read that as settled because the 2026-08-12 concurrent-legs run had also scored 27 - 'n=2 agreement'. Two further sequential runs came in at 30 and 29. The 27 was correct for the run that produced it; what does not survive is treating a pair of agreeing draws as a measurement of the rung. Both 27s sit at the bottom of the range this configuration actually spans. *** WHERE THE MOVEMENT IS, AND WHAT THAT IS WORTH *** The three totals span 3 points, but the per-repo legs span more: repo 1 scored 8, 14 and 13 of 45, repo 2 scored 19, 16 and 16 of 60. The repo-1 range alone is 6 points on a 45-bug repo, wider than the range of the totals it feeds. The legs moved in opposite directions and the totals landed close together. That is an OBSERVATION about three runs and nothing more - it is not a mechanism, and it is not a claim that totals are steadier than legs. Three draws cannot establish that, and this pattern arises from chance often enough to be unremarkable. *** RECEIPTS *** All six legs exited 0. Per-leg (repo 1 / repo 2): wall 19.1/22.0, 18.8/23.5 and 23.9/21.9 minutes; floor 6.84/10.12, 9.38/8.59 and 13.57/7.30 dollars. Genuine extras 16, 16 and 22. Claimed-only 1, 0 and 1, all on repo 1. COST IS A FLOOR, NOT A BILL: the grok CLI reports context occupancy rather than cumulative usage, so every grok row on this board is reconstructed from the CLI's own session logs at xAI list rates. The three floors are 16.96, 17.96 and 20.88 dollars and this row shows their mean. That spread is not pricing noise - the reconstruction scales with how many inference requests a run made (106, 118 and 128 across the three), so it tracks real differences in how much work the agent did. SEQUENTIAL legs throughout - one repo at a time, one machine, one key - which is what 'seq' means in the name and why this wall is comparable to the rest of the board. Effort verified active by a fresh zero-token ACP readback before each launch. Judge: Codex gpt-5.5 on all six legs.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 29, 30, 33, 34, 37, 39, 40, 41, 43, 44, 45, 47, 49, 50, 52, 60, 62, 66, 70, 71, 74, 76, 77, 82, 83, 84, 89, 91, 92, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 3, "20": 3, "21": 1, "22": 2, "23": 2, "25": 1, "29": 3, "30": 3, "33": 2, "34": 2, "37": 1, "39": 1, "40": 2, "41": 1, "43": 3, "44": 1, "45": 2, "47": 3, "49": 3, "50": 3, "52": 1, "60": 3, "62": 3, "66": 1, "70": 1, "71": 3, "74": 1, "76": 3, "77": 2, "82": 1, "83": 3, "84": 3, "89": 1, "91": 1, "92": 3, "98": 3, "99": 3, "100": 3, "102": 3 }, "coverage_runs": 3, "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 101 }, { "id": "Grok 4.7 (xhigh) seq - mean of 3", "n_runs": 3, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 28.7, "repo1_fixed": 13.3, "repo2_fixed": 15.3, "partial": 1.7, "claimed_only": 0.3, "extras": 31.7, "false_positive_fixes": 0, "wall_min": 46.2, "cost_usd": 23.17, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-21 by the n=4 row at the same configuration (28.8). A fourth run scored 29 of 105 and moved the mean by +0.1 of a bug, so nothing this row claimed is retracted - it said the generation step moved nothing that is being counted, and a fourth draw said the same on more evidence. What the newer row can say that this one could not: the wall comparison, which this row published as undecidable because only one of its three runs was uncontended. The fourth run was also uncontended, and the two clean runs put this model within about a minute and a half of Grok 4.6. Kept rather than deleted because it is the receipt for what the board showed for part of 2026-09-21.", "caveat": "A mean of three runs, not a single measurement; the three span 25 to 31 of 105 against 4.6's 27 to 30, so this tie rests on the wider of the two distributions. Cost uses a per-session reconstruction the other grok rows do not yet use; the note gives both figures and the gap holds either way. The wall is contended upward on four of the six legs and is not a basis for a speed comparison. The three runs spawned different numbers of subagent sessions on identical arm definitions, which the board's regime check cannot verify on this harness.", "note": "Grok 4.7 at xhigh through xAI's own CLI, the first 4.7 row on this board, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 25, 30 and 31 of 105 (repo 1: 12, 12, 16 - repo 2: 13, 18, 15). Mean 28.7, range 6. *** THE HEADLINE IS THAT NOTHING MOVED *** Grok 4.6 at the SAME tier, also a mean of three, stands at 28.7 of 105. This row is 28.7. Three runs a side at a verified ceiling, and the totals are identical to the decimal - the cleanest null result this board has produced for a generation step. It is published as one. A tie is a measurement, and publishing only when a number improves would make this a scoreboard for vendors rather than a benchmark. *** THE IDENTICAL TOTAL HIDES A SWAP *** Repo 1 scored 12, 12, 16 of 45 against 4.6's mean of 11.7; repo 2 scored 13, 18, 15 of 60 against 17.0. Up about a point and a half on one codebase, down about the same on the other. Both movements sit inside the spread of the runs that produced them, so this is a direction worth testing and not a finding. *** WHAT DOES MOVE, AND IT MOVES THE WRONG WAY: COST *** Reconstructed floors of $16.54, $23.04, $29.94 for a mean of $23.17, against $16.89 for the three 4.6 runs on the SAME arithmetic - about 37% more for the same score. READ THE COST COLUMN CAREFULLY: the reconstruction used to take the largest context in a run and charge every request in that run against it, which overstates any run that fans out into subagents by up to 25%, and it now measures each session separately. This row carries the corrected figure. The 4.6 row still shows $18.60 on the old arithmetic and has not yet been restated; on that same old arithmetic these three runs come to $26.18. So the gap is about a third either way. What is inconsistent across the board right now is the arithmetic, not the finding. Both numbers are floors twice over: the grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates - the same 2/0.5/6 per MTok basis, and xAI prices 4.7 identically to 4.6 - and xAI bills a request whose prompt reaches 200K at DOUBLE those rates, which these runs did, by an amount this method cannot see. *** AND WHAT MOVES IN 4.7'S FAVOUR: BUGS NOBODY PLANTED *** Genuine extras - real defects found in the repo that are not part of the planted set - ran 26, 35, 34 across the three runs, a mean of 31.7, against 18.0 for 4.6: roughly 76% more. The most defensible reading of this wave is that 4.7 finds materially more real bugs than 4.6 and no more of the ones being counted. Whether that is a better bug-hunter or a worse instruction-follower is a judgment the number does not settle. *** PER BUG, THE TWO MODELS ARE NEARLY THE SAME MODEL *** Across three runs a side, on repo 1 each touched 18 distinct planted bugs with 15 shared; on repo 2 it was 21 against 22 with 18 shared. There is not one bug on either repo that 4.6 found on all three runs and 4.7 never found - zero hard regressions - and exactly one that 4.7 found on all three and 4.6 never did. Every other bug unique to one model was a single-run hit: noise, not capability. Consistency shifted in opposite directions per repo: 4.7 holds 10 bugs on all three runs against 4.6's 5 on repo 1, and 10 against 14 on repo 2. *** THE RUNS DID NOT BEHAVE ALIKE, AND THE AUTOMATIC CHECK CANNOT SEE IT HERE *** The regime check that refuses a mean over mismatched configurations parses Claude Code transcripts, so it returns NOT CHECKED for every grok row on this board. The CLI's own session logs show what it would have looked for. Sessions spawned per leg (repo 1 / repo 2) - this model: run 1 1 and 1; run 2 1 and 3; run 3 5 and 1. The 4.6 rows at the same tier: run 1 3 and 3; run 2 4 and 3; run 3 5 and 3. 4.6 never once ran a leg as a single session; this model did it four times out of six, and when it does fan out it commits far harder - 184 and 206 inference requests against 4.6's 88 to 128. Fan-out tracks COST almost perfectly across these runs and tracks SCORE not at all: 4.6's weakest repo-1 leg fanned out to three sessions, and the two single-session repo-1 legs here scored the same as each other. The arm definitions are byte-identical clones, so this is emergent agent behaviour rather than configuration - the same class of hidden difference the regime check was built for. An OBSERVATION over three runs, not a mechanism. *** THE TIER IS THE MODEL CEILING AND THAT WAS CHECKED, NOT ASSUMED *** A zero-token ACP readback before each launch returned reasoning_effort xhigh for a requested xhigh, and the enum grok-4.7 advertises is low, medium, high, xhigh with high as the CLI default - nothing above xhigh exists to ask for. The grok CLI silently clamps an unknown effort string, so an echo is the only thing that makes this label honest. The served model is named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. *** RECEIPTS *** All six legs exited 0. Per-run wall 39.6, 47.3, 51.8 minutes against 43.1 for the 4.6 mean - but that comparison is NOT decidable from this wave: only run 1 had the machine to itself, runs 2 and 3 overlapped each other, and contention biases a wall upward. On the one uncontended run this was the fastest run at this tier the board has recorded. Legs SEQUENTIAL within every run, one repo at a time - that is what seq means here. Judge: blind Codex gpt-5.5 on all six legs, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 27, 28, 29, 30, 33, 34, 37, 38, 39, 40, 43, 45, 47, 49, 50, 57, 60, 62, 70, 71, 76, 82, 83, 84, 89, 91, 92, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "22": 2, "23": 1, "25": 3, "27": 1, "28": 1, "29": 3, "30": 3, "33": 1, "34": 2, "37": 3, "38": 1, "39": 3, "40": 1, "43": 3, "45": 3, "47": 3, "49": 2, "50": 2, "57": 1, "60": 3, "62": 3, "70": 1, "71": 3, "76": 3, "82": 1, "83": 3, "84": 2, "89": 1, "91": 2, "92": 3, "95": 3, "97": 1, "98": 1, "99": 3, "100": 3, "102": 2 }, "coverage_runs": 3, "turns": 506, "turns_repo1": 181, "turns_repo2": 325, "group_rank": 1 }, { "id": "GPT-6.1 Sol (medium effort)", "n_runs": 1, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-29", "color": "#2BA580", "fixed": 28, "repo1_fixed": 16, "repo2_fixed": 12, "partial": 3, "claimed_only": 0, "extras": 32, "false_positive_fixes": 0, "wall_min": 34.0, "cost_usd": 1.85, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 2 (29.0 of 105). This row reported 28 of 105 from a single run. Its note announced two replicates; this tier was then cut to one, so the mean is of 2. The replicate is a byte-identical re-run; across 2 the runs span 28 to 30. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat 28 as one draw until the replicates land. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates (Codex caps the context for this model below 272K). Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at medium effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 28 of 105: 16 of 45 on repo 1 and 12 of 60 on repo 2, 32 genuine extras, 34.0 minutes, $1.85, 95 turns (53 and 42). ONE RUN - two replicates follow and will supersede this row. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at high 38, GPT-6.1 Sol at xhigh 41, GPT-6.1 Sol at max 44. At the same tier the board publishes GPT-6 Sol at medium 14, GPT-6 Astra at medium 34, Sonnet 5.5 at medium 23.3 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at medium. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 35, 39, 40, 41, 43, 44, 45, 47, 49, 50, 76, 82, 83, 84, 92, 97, 98, 99, 100 ], "turns": 95, "turns_repo1": 53, "turns_repo2": 42, "group_rank": 103 }, { "id": "Qwen3.8-Max (max effort, Alibaba API)", "n_runs": 1, "model": "Qwen3.8-Max", "vendor": "Alibaba", "local": false, "harness": "Claude Code / Alibaba API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-11", "color": "#E8A020", "fixed": 28, "repo1_fixed": 13, "repo2_fixed": 15, "partial": 2, "claimed_only": 3, "extras": 6, "false_positive_fixes": 0, "wall_min": 103.4, "cost_usd": 26.29, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Qwen3.8-Max (max effort, Alibaba API) - mean of 3", "caveat": "One run at this tier. The effort ceiling is real but flat: the same probe shows high (9946-14061), xhigh (8266-19309) and max (6542-14902) are mutually indistinguishable with means that fall as the nominal tier rises, so max is the top of the dial and buys nothing measurable over high - the same two-rung shape the Qwen3.8-Flash probe found, which suggests it is a property of Alibaba's serving rather than of either model. Note also that this endpoint's accept-list CONTRADICTS the QwenCloud doc the superseded row relied on (low|medium|xhigh with high->xhigh), which is one more reason that row's tier claim was never safe. Cost is a list-price ESTIMATE and not a bill - DashScope exposes no credits API on this path - computed from the run's own token counts at 2.00 in / 2.50 cache-write / 0.25 cache-read / 6.00 out per MTok, from OpenRouter's Alibaba-endpoint listing. Finally, do NOT read the 9-point gain as caused by the path. It is n=1 against n=1 and almost all of it is one repo - repo 1 went 5/45 to 13/45 while repo 2 moved by one - and this board has seen the same model at the same setting swing 6 points on repo 1 (Opus 5 at medium: 11/8/14). A systematic path effect should have moved both repos and did not, so the score difference is unexplained and may simply be variance. What the pair DOES establish beyond variance is the cost and token behaviour, where the gap is ~5,000x and not a coin flip.", "note": "The Aug 3 Qwen3.8-Max row re-run on Alibaba's OWN Anthropic-compatible endpoint - no local proxy, no aggregator - and it is the clearest demonstration on this board that a row records a SERVING PATH and not just a model. Same model, same harness, same two repos, same laptop, one hop removed: 28/105 against 19/105, 103.4 min against 148.1, $26.29 against $33.30. Nine points more for a fifth less money and a third less time, and the gain is concentrated in repo 1, which went 5/45 to 13/45. The mechanism is visible in the token columns. The OpenRouter run billed 12.94% of its prompt tokens as UNCACHED INPUT - 8,282,777 of them, $16.57, half that row's cost - while this one billed 0.0019%, or 1,716 tokens across both legs. Caching that keeps working means the model keeps its context instead of rebuilding it, which is why output ROSE 23% (168,395 tokens against 136,426) while wall time FELL: it did more thinking and less re-reading. Effort here is measured, not requested - the endpoint 400-rejects an invented tier and names its accept-list (low, medium, high, xhigh, max), and an n=3-per-tier probe on this exact path separates the bottom of the dial cleanly (mean output tokens low=4495 against high=12422, low's range overlapping none of the top three, 3.09x between-tier ratio against a 2.34x worst within-tier spread). 28 strict matches, 2 partials, 3 claimed-only, 6 genuine unplanted extras.", "fixed_bugs": [ 13, 20, 22, 25, 26, 27, 29, 30, 39, 40, 41, 43, 44, 47, 49, 50, 59, 60, 61, 71, 76, 78, 83, 92, 97, 99, 100, 102 ], "turns": 398, "turns_repo1": 126, "turns_repo2": 272, "group_rank": 0 }, { "id": "GPT-6 Astra (low effort)", "n_runs": 1, "model": "GPT-6 Astra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-09-05", "color": "#1B9E77", "fixed": 27, "repo1_fixed": 18, "repo2_fixed": 9, "partial": 0, "claimed_only": 0, "extras": 25, "false_positive_fixes": 0, "wall_min": 32.1, "cost_usd": 11.69, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 22, 25, 26, 29, 30, 33, 34, 35, 36, 39, 40, 41, 43, 44, 45, 49, 50, 68, 77, 83, 98, 99, 100, 102 ], "turns": 92, "turns_repo1": 49, "turns_repo2": 43, "group_rank": 4 }, { "id": "Grok 4.6 (xhigh) seq", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified", "date": "2026-08-28", "color": "#757A82", "fixed": 27, "repo1_fixed": 8, "repo2_fixed": 19, "partial": 2, "claimed_only": 1, "extras": 16, "false_positive_fixes": 0, "wall_min": 41.1, "cost_usd": 16.96, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same configuration (28.7, rounding to 29). This row is ONE of three sequential runs that scored 27, 30 and 29, and it is the lowest. ONE READING IN THE NOTE BELOW IS RETRACTED: that this run and the 2026-08-12 concurrent run both scoring 27 was 'n=2 agreement' establishing the number. Two agreeing draws are not a measurement of a rung; two further runs put the mean at 28.7 with both 27s at the bottom of the observed range. The 27 itself stands - it is what this run scored. Kept rather than deleted because it is the receipt for what the board published on 2026-08-28, and a retraction is only legible beside its claim.", "caveat": null, "note": "Sequential re-run of the 08-12 concurrent-legs xhigh - both scored 27/105 (n=2 agreement), and this one has a comparable single-machine wall, so it replaces the dotted original.", "fixed_bugs": [ 13, 16, 20, 29, 30, 39, 43, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 82, 83, 84, 89, 91, 92, 98, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 101 }, { "id": "Grok 4.6 (xhigh)", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "xhigh", "effort_status": "verified_ceiling", "date": "2026-08-12", "color": "#757A82", "fixed": 27, "repo1_fixed": 10, "repo2_fixed": 17, "partial": 2, "claimed_only": 0, "extras": 15, "false_positive_fixes": 0, "wall_min": 34.4, "cost_usd": 22.73, "cost_kind": "floor", "wall_note": "Both repo legs ran concurrently on one machine and one key; contention can inflate this figure, not shrink it.", "ceiling": true, "featured": false, "superseded": "Sequential re-run on 2026-08-28 (also 27/105) supersedes this concurrent-legs run for a comparable wall.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 25, 29, 30, 33, 34, 37, 43, 47, 49, 50, 52, 60, 62, 71, 76, 77, 83, 84, 91, 92, 98, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 101 }, { "id": "Opus 5 (max effort)", "n_runs": 1, "model": "Opus 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-08-01", "color": "#E17757", "fixed": 27, "repo1_fixed": 13, "repo2_fixed": 14, "partial": 2, "claimed_only": 5, "extras": 2, "false_positive_fixes": 0, "wall_min": 60.0, "cost_usd": 54.94, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 21, 22, 25, 27, 29, 30, 33, 34, 39, 43, 45, 47, 49, 60, 70, 71, 76, 82, 83, 84, 91, 92, 97, 98, 99 ], "turns": 295, "turns_repo1": 148, "turns_repo2": 147, "group_rank": 300 }, { "id": "Grok 4.7 (medium) - mean of 3", "n_runs": 3, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "medium", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 26.7, "repo1_fixed": 12, "repo2_fixed": 14.7, "partial": 2.4, "claimed_only": 0, "extras": 22.7, "false_positive_fixes": 0, "wall_min": 30.7, "cost_usd": 16.76, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration spanning 23 to 30 of 105. The Grok 4.6 row at this rung is a SINGLE run, so that comparison is three draws against one and is not a comparison of two distributions. Cost is a reconstructed FLOOR rather than a bill, and uses a per-session arithmetic the 4.6 rows on this board do not yet use; the note gives both figures. The runs spawned different numbers of subagent sessions on identical arm definitions, which the board's regime check cannot verify on this harness.", "note": "Grok 4.7 at medium through xAI's own CLI, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 30, 23 and 27 of 105 (repo 1: 11, 11, 14 of 45 - repo 2: 19, 12, 13 of 60). Mean 26.7, spread 7. This supersedes the single-run row published earlier the same day at 30 of 105, which said the same thing on one draw. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 at medium scored 23 of 105 (7 on repo 1, 16 on repo 2) - A SINGLE RUN, so the samples are not the same size and the comparison is three draws against one. Difference +3.7. That 4.6 number falls INSIDE the spread of these three runs, which is the clearest way to say the two are not distinguishable on this evidence. *** WHERE THIS RUNG SITS ON ITS OWN DIAL *** The same model at xhigh is 28.8 of 105 over four runs; this rung is 26.7 over three, a gap of +2.1. The dial as the board now shows it: low 11 (n=1), medium 26.7 (n=3), high 19.3 (n=3), xhigh 28.8 (n=4). For comparison Grok 4.6's dial ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, each of those from a single run. Whether that flatness is a property of the dial or a coincidence of two single draws is exactly what replicating these rungs was for. *** WHAT THE SPREAD ITSELF SAYS *** 7 points between the best and worst run of an IDENTICAL configuration - the three arms are clones and differ only in name. Any single-run number at this rung, including the one this row replaces, was carrying that much slack without being able to show it. Read any n=1 row on this board, from any vendor, with that in mind. *** COST AND TIME *** Reconstructed floors of $25.15, $9.90, $15.25 for a mean of $16.77; walls 32.4, 27.1, 32.6 minutes, mean 30.7. COST IS A FLOOR, not a bill: the grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates on the same 2/0.5/6 per MTok basis, and xAI bills a request whose prompt reaches 200K at DOUBLE those rates. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic that the 4.6 rows on this board still use it would read $18.40. Compare cost across rows only where both use the same arithmetic. *** GENUINE EXTRAS *** Real defects found in the repo that are not part of the planted set ran 30, 16 and 22 across the three runs, a mean of 22.7. Extras are not scored; they are reported because a model that finds real bugs nobody planted is doing something the planted-bug total cannot see. *** THE TIER WAS CHECKED, NOT ASSUMED, AND AT THE LOWER RUNGS THAT MATTERS MOST *** A zero-token ACP readback before each of these three launches returned reasoning_effort medium for a requested medium, with the served model named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except that readback - the run would be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run, one repo at a time, and only one grok run on the machine at a time - two grok agents share one login. Sessions spawned per leg (repo 1 / repo 2): run 1 3 and 5; run 2 1 and 1; run 3 4 and 1. That count varies run to run on identical arm definitions for this model, tracks cost closely, and has tracked score not at all. The board's automatic regime check parses Claude Code transcripts, so it returns NOT CHECKED for every grok row here and the session counts above are the manual substitute. Judge: blind Codex gpt-5.5 on all six legs, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 27, 28, 29, 30, 33, 34, 37, 38, 39, 43, 45, 47, 49, 50, 56, 57, 60, 62, 71, 76, 78, 82, 83, 84, 87, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 3, "20": 3, "21": 1, "22": 3, "25": 3, "27": 1, "28": 1, "29": 2, "30": 3, "33": 1, "34": 1, "37": 3, "38": 1, "39": 1, "43": 3, "45": 3, "47": 3, "49": 3, "50": 2, "56": 1, "57": 1, "60": 3, "62": 3, "71": 3, "76": 3, "78": 1, "82": 1, "83": 3, "84": 3, "87": 1, "92": 2, "97": 1, "98": 2, "99": 3, "100": 3, "102": 2 }, "coverage_runs": 3, "turns": 417, "turns_repo1": 165, "turns_repo2": 252, "group_rank": 3 }, { "id": "Qwen3.8-Flash (max effort)", "n_runs": 1, "model": "Qwen3.8-Flash", "vendor": "Alibaba", "local": false, "harness": "Claude Code / Alibaba API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-11", "color": "#F5D49B", "fixed": 26, "repo1_fixed": 13, "repo2_fixed": 13, "partial": 3, "claimed_only": 3, "extras": 7, "false_positive_fixes": 0, "wall_min": 97.3, "cost_usd": 1.81, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "One run at this tier, and repo 1 is a RETRY. The first repo-1 attempt at this exact setting made 70 successful tool calls across 73 turns with zero permission denials, then wrote 'Let me check a few remaining details before fixing...' and ended its turn having written no report - a healthy environment and a model that narrated its next step instead of taking it. The retry delivered 13/45 in 42.5 minutes, so that was a fluke rather than a property, but a top-effort run that silently stops on one repo in two attempts is itself a reliability signal, and the published wall and cost figures cover the retry only. Cost is a list-price ESTIMATE and not a bill - DashScope exposes no credits API on this path - computed from the run's own token counts at 0.15 in / 0.20 cache-write / 0.016 cache-read / 0.47 out per MTok. Do not read this row as a ceiling for the Qwen 3.8 family: it is the ceiling of the SMALL model. The Qwen3.8-Max row on this board ran a different path entirely (Claude Code -> local proxy -> OpenRouter) and is not a clean comparison.", "note": "The top of Alibaba's own accept-list on Alibaba's own Anthropic-compatible endpoint - no aggregator, no local proxy. 26/105 for $1.81 at list. Read it against its low companion, which is the point of running both: max buys THREE points over low (23/105) for 32% more money and the same wall clock (97.3 min against 98.1). The effort label is measured rather than requested - the endpoint 400-rejects an invented tier and names its accept-list (low, medium, high, xhigh, max), and an n=3-per-tier magnitude probe on this exact path shows the dial binds at the bottom: mean output tokens run low=3557, medium=4579, high=19329, and the 5.43x spread between lowest and highest mean clears the worst within-tier spread of 2.00x. So max is the real ceiling of a real dial. What it is NOT is a fifth rung: high (13933-27933), xhigh (12523-24778) and max (13767-19431) overlap almost entirely and their means FALL as the nominal tier rises, so paying for max over high buys nothing measurable. 26 strict matches, 3 partials, 3 claimed-only, 7 genuine unplanted extras.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 26, 27, 29, 30, 39, 40, 43, 47, 50, 60, 62, 76, 77, 83, 84, 91, 92, 97, 98, 99 ], "turns": 363, "turns_repo1": 122, "turns_repo2": 241, "group_rank": 300 }, { "id": "Opus 5 (xhigh effort)", "n_runs": 1, "model": "Opus 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-08-28", "color": "#E17757", "fixed": 26, "repo1_fixed": 14, "repo2_fixed": 12, "partial": 1, "claimed_only": 4, "extras": 3, "false_positive_fixes": 0, "wall_min": 49.9, "cost_usd": 63.25, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": "The Opus dial is nearly flat at the top: high 21, medium 24, xhigh 26, max 27.", "fixed_bugs": [ 13, 16, 20, 21, 22, 25, 27, 29, 30, 33, 34, 39, 43, 45, 49, 50, 61, 70, 71, 76, 83, 84, 92, 97, 98, 99 ], "turns": 319, "turns_repo1": 134, "turns_repo2": 185, "group_rank": 301 }, { "id": "Qwen3.8-Max (max effort, Alibaba API) - mean of 3", "n_runs": 3, "model": "Qwen3.8-Max", "vendor": "Alibaba", "local": false, "harness": "Claude Code / Alibaba API", "effort": "max", "effort_status": "first_party", "date": "2026-09-15", "color": "#E8A020", "fixed": 25.7, "repo1_fixed": 12.3, "repo2_fixed": 13.3, "partial": 1.4, "claimed_only": 2, "extras": 7, "false_positive_fixes": 0, "wall_min": 117.1, "cost_usd": 26.76, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Superseded row: the single 28/105 run published 2026-09-11 stays on the board, marked superseded, not deleted - it is the best of the three, and the mean is the number the board leads with per the standing rule that a rung measured three times publishes at n=3.", "note": "Runs scored 28, 24 and 25 of 105 (repo 1: 13, 11, 13 - repo 2: 15, 13, 12). Mean 25.7, range 4. This replication moves the row DOWN, not up: the single run published 2026-09-11 was the best of the three (28), and the mean sits 2.3 points below it - the opposite of the Qwen3.8-27B replication, which held steady on its mean. Regime check: SAME REGIME (gbvb); SAME REGIME (abhb). First-party max effort, Alibaba's own DashScope endpoint, native precision, 1,000,000 declared context - same endpoint and declaration as the superseded single run. Cost quoted here is the MEAN per-run cost across the three legs (26.29, 29.85, 24.15), not their sum; the three runs together cost 80.29 dollars, all list-rate estimates - this endpoint publishes no usage API, so none of these figures are a real bill.", "fixed_bugs": [ 13, 16, 20, 22, 25, 26, 27, 28, 29, 30, 32, 39, 40, 41, 43, 44, 45, 47, 49, 50, 59, 60, 61, 62, 71, 76, 78, 79, 83, 84, 92, 97, 99, 100, 102 ], "bug_hits": { "13": 3, "16": 1, "20": 3, "22": 1, "25": 3, "26": 3, "27": 3, "28": 1, "29": 3, "30": 3, "32": 1, "39": 2, "40": 2, "41": 1, "43": 3, "44": 2, "45": 2, "47": 3, "49": 3, "50": 3, "59": 1, "60": 3, "61": 1, "62": 2, "71": 3, "76": 3, "78": 1, "79": 1, "83": 3, "84": 2, "92": 1, "97": 3, "99": 3, "100": 1, "102": 3 }, "coverage_runs": 3, "turns": 373, "turns_repo1": 136, "turns_repo2": 236, "group_rank": 0 }, { "id": "GPT-6 Sol (xhigh effort)", "n_runs": 1, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 25, "repo1_fixed": 10, "repo2_fixed": 15, "partial": 1, "claimed_only": 0, "extras": 24, "false_positive_fixes": 0, "wall_min": 51.4, "cost_usd": 7.67, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; it does compare cleanly to the other GPT-6 Sol rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs, so the wall figure is not contended.", "note": "OpenAI's GPT-6 Sol at xhigh effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 25 of 105: 10 of 45 on repo 1 and 15 of 60 on repo 2, 24 genuine extras, 51.4 minutes, $7.67. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Sol ladder currently reads low 6, medium 14, high 20, xhigh 25, max 29.3 (n=3). For comparison the board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max and 43 at xhigh, and GPT-5.6 Sol at 43.5 (mean of 2) at max. This model's max rung replicated to three runs and landed below all of them, which is why the rungs beneath it are being measured at all: a sweep separates a weak model from a mis-set dial. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration, and this model's own max rung spans three runs. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-sol, same pinned executable, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows. The GPT-6 Sol rungs DO compare cleanly to each other - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for both legs, so these wall figures compare directly against other uncontended rows. (Corrected 2026-09-23: this row first published saying the run shared the laptop, which its receipts do not support - neither leg recorded a peer. The claim was a literal copied from a publish script, and the wall column is the evidence that the effort dial reaches the model at all, so it is worth getting right in both directions.)", "fixed_bugs": [ 13, 16, 17, 20, 22, 25, 29, 30, 41, 45, 47, 49, 62, 71, 76, 77, 80, 82, 83, 84, 92, 98, 99, 100, 102 ], "turns": 260, "turns_repo1": 107, "turns_repo2": 153, "group_rank": 201 }, { "id": "DeepSeek V4.1 Flash (max effort)", "n_runs": 1, "model": "DeepSeek V4.1 Flash", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / DeepSeek API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-10", "color": "#5A3EC8", "fixed": 24, "repo1_fixed": 14, "repo2_fixed": 10, "partial": 0, "claimed_only": 0, "extras": 7, "false_positive_fixes": 0, "wall_min": 42.6, "cost_usd": 1.08, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row at the same rung (21.7/105). This row is ONE run of three and it is the HIGHEST of the three: identical runs scored 24, 19 and 22. Nothing in it is retracted - the 24 happened and this is its receipt - but it was the configuration's best day, so any comparison drawn against 24 was drawn against that. Kept rather than deleted because a supersession is only legible next to the row it supersedes.", "caveat": "The endpoint deserializer also accepts an undocumented `ultra`, which DeepSeek own docs never mention (they give none|low|high|max, default high). Sampled n=4 it straddles both high and max and resolves to neither, and nothing in that sample exceeds max band - so `max` is the top of the documented dial and nothing measured reasons more than it, which is what the ceiling label claims. Which tier `ultra` aliases is unresolved and left that way rather than guessed.", "note": "DeepSeek newest Flash generation, benchmarked on DeepSeek OWN Anthropic-compatible API - no aggregator, no proxy. Claude Code puts --effort on the wire as output_config.effort; the endpoint 400-rejects an invented tier and names its enum, and a thinking-volume sweep separates the tiers with DISJOINT ranges (high mean 30.5K chars, max mean 83.6K, n=3), so max was applied and not merely requested. Cost is a real bill, not an estimate: DeepSeek sells prepaid credit, so each leg is charged at the balance drop across it. $1.08 of real credit for 24/105 - the same score as Fable 5 and as Opus 5 at medium. Those two are list estimates and this is a bill, so read the gap as an order of magnitude and not as a ratio.", "fixed_bugs": [ 13, 20, 21, 22, 23, 25, 27, 28, 29, 30, 33, 34, 43, 45, 50, 60, 70, 76, 83, 84, 92, 97, 98, 99 ], "turns": 485, "turns_repo1": 193, "turns_repo2": 292, "group_rank": 0 }, { "id": "Opus 5.5 (low effort) - mean of 2", "n_runs": 2, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 24, "repo1_fixed": 12, "repo2_fixed": 12, "partial": 1, "claimed_only": 2.5, "extras": 9.5, "false_positive_fixes": 0, "wall_min": 11.3, "cost_usd": 8.73, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (22.3 of 105). This row reported 24.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 19 to 25. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 23 and 25 of 105 - spread 2. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at low effort through Claude Code on the first-party API, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 23 and 25 of 105 (repo 1: 11, 13 of 45 - repo 2: 12, 12 of 60). Mean 24.0, spread 2. *** THIS SUPERSEDES OPUS 5.5 (LOW EFFORT) (23 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 12.4, 10.2 minutes, mean 11.3; cost $8.68, $8.79, mean $8.73. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 14 and 5 across the runs, mean 9.5 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 26, 29, 30, 33, 34, 39, 40, 43, 44, 45, 47, 49, 50, 62, 71, 76, 77, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 2, "20": 2, "22": 1, "23": 1, "25": 1, "26": 1, "29": 2, "30": 2, "33": 2, "34": 2, "39": 2, "40": 1, "43": 2, "44": 1, "45": 1, "47": 2, "49": 2, "50": 1, "62": 1, "71": 2, "76": 2, "77": 1, "83": 2, "84": 2, "92": 1, "97": 1, "98": 1, "99": 2, "100": 2, "102": 2 }, "coverage_runs": 2, "turns": 136, "turns_repo1": 58, "turns_repo2": 78, "group_rank": 204 }, { "id": "Opus 5 (medium effort)", "n_runs": 1, "model": "Opus 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-08-27", "color": "#E17757", "fixed": 24, "repo1_fixed": 11, "repo2_fixed": 13, "partial": 2, "claimed_only": 4, "extras": 4, "false_positive_fixes": 0, "wall_min": 30.5, "cost_usd": 37.4, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 21, 22, 27, 30, 33, 34, 39, 43, 45, 47, 50, 57, 60, 71, 76, 82, 83, 84, 92, 97, 99, 100 ], "turns": 260, "turns_repo1": 107, "turns_repo2": 153, "group_rank": 303 }, { "id": "Fable 5", "n_runs": 1, "model": "Fable 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-07-26", "color": "#DC603A", "fixed": 24, "repo1_fixed": 9, "repo2_fixed": 15, "partial": 0, "claimed_only": 1, "extras": 3, "false_positive_fixes": 0, "wall_min": 31.4, "cost_usd": 74.52, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 20, 21, 22, 25, 27, 39, 43, 45, 47, 49, 50, 60, 62, 71, 76, 77, 82, 83, 92, 98, 99, 100, 102 ], "turns": 244, "turns_repo1": 110, "turns_repo2": 134, "group_rank": 102 }, { "id": "MiMo-V2.6-Flash - mean of 3", "n_runs": 3, "model": "MiMo-V2.6-Flash", "vendor": "Xiaomi", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-22", "color": "#9898D7", "fixed": 23.3, "repo1_fixed": 8, "repo2_fixed": 15.3, "partial": 1.7, "claimed_only": 3.4, "extras": 7.3, "false_positive_fixes": 0, "wall_min": 70.7, "cost_usd": 0.49, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 19, 25 and 26 of 105 - spread 7 - so read the mean with that width. The comparison against the Pro model is PARITY, not a win: the two means are inside each other's spread, and Flash is actually weaker on repo 1 and stronger on repo 2, so the overall match is an average of two opposite tilts. Cost is a real OpenRouter bill. Effort is `default`: the reasoning dial was measured inert on the Pro model but never probed on this one, so this is not a measurement at any asserted reasoning level. Wall figures are contended and biased upward.", "note": "Xiaomi's MiMo-V2.6-Flash - the cheap sibling released alongside the Pro model - driven by Claude Code through an Anthropic-API shim onto OpenRouter, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 19, 25 and 26 of 105 (repo 1: 8, 7, 9 of 45 - repo 2: 11, 18, 17 of 60). Mean 23.3, spread 7. Run as n=3 from the start rather than promoted to a mean later. *** THE CHEAP MODEL MATCHES THE FLAGSHIP, AND THAT IS THE RESULT *** the Pro model's mean of three on this same board, same harness, same route is 22.7 of 105. Flash's 23.3 is nominally higher at roughly half the cost per run ($0.49 against $0.86). Both means sit comfortably inside the other's run-to-run spread, so the claim this supports is PARITY, not victory: on this benchmark Xiaomi's cheap model is not measurably worse than its flagship. Read it that way rather than as a ranking. *** THE PARITY IS AN AVERAGE OF TWO OPPOSITE TILTS, NOT A UNIFORM MATCH *** Flash is weaker than Pro on repo 1 (8, 7, 9 of 45, against Pro's 9, 7 and 8) and stronger on repo 2 (11, 18, 17 of 60, against Pro's 11, 15 and 14). A single-repo benchmark would have ranked these two models in opposite orders depending on which repo it used. *** WALL AND COST *** Walls 66.6, 81.1, 64.3 minutes, mean 70.7; cost $0.44, $0.59, $0.45, mean $0.49. COST IS A REAL BILL, the OpenRouter credits delta across each run's legs, comparable to other billed rows here and not to list-priced or floor-reconstructed ones. *** GENUINE EXTRAS *** 5, 8 and 9 across the three runs, mean 7.3 - real defects found in the repos that are not part of the planted set, and consistently more than the Pro model found. *** THE DIAL *** effort is `default`. This route's `reasoning` field was measured inert on the Pro model (n=3 per condition, ranges fully overlapping); that probe was run on Pro, NOT on Flash, so for this model the dial is untested rather than known-inert. None of these three runs asserted a reasoning state. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run, one repo at a time, and the three runs ran one at a time in order. Judge: blind Codex gpt-5.5 on all six legs - neither this model nor a sibling of it - with no judge switching anywhere in this mean. The runs shared the laptop with legs from other vendors, which the row's wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 25, 26, 27, 30, 33, 34, 39, 43, 44, 47, 50, 51, 60, 61, 62, 66, 69, 70, 71, 76, 77, 79, 80, 82, 83, 84, 91, 92, 96, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 3, "20": 3, "25": 2, "26": 1, "27": 1, "30": 3, "33": 1, "34": 1, "39": 2, "43": 3, "44": 3, "47": 1, "50": 2, "51": 1, "60": 1, "61": 1, "62": 2, "66": 1, "69": 1, "70": 2, "71": 3, "76": 3, "77": 1, "79": 2, "80": 1, "82": 2, "83": 3, "84": 1, "91": 1, "92": 2, "96": 1, "97": 3, "98": 3, "99": 3, "100": 3, "102": 2 }, "coverage_runs": 3, "turns": 415, "turns_repo1": 132, "turns_repo2": 283, "group_rank": 106 }, { "id": "Sonnet 5.5 (medium effort) - mean of 3", "n_runs": 3, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-30", "color": "#E68F74", "fixed": 23.3, "repo1_fixed": 8.7, "repo2_fixed": 14.7, "partial": 2, "claimed_only": 2.7, "extras": 8.7, "false_positive_fixes": 0, "wall_min": 17.4, "cost_usd": 11.11, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 18, 28 and 24 of 105 - spread 10. Read the mean with that width. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. Wall figures are contended.", "note": "Anthropic's Sonnet 5.5 at medium effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 18, 28 and 24 of 105 (repo 1: 8, 12, 6 - repo 2: 10, 16, 18). Mean 23.3, range 10. *** THIS SUPERSEDES SONNET 5.5 (MEDIUM EFFORT) (18 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at medium effort 30.3 (mean of 3) of 105. *** WALL AND COST *** walls 11.9, 18.4, 21.9 minutes, mean 17.4; cost $8.39, $8.94, $16.02, mean $11.12. *** GENUINE EXTRAS *** 8, 7 and 11 across the runs, mean 8.7, never folded into the score. *** HOW IT SPENT THE RUN *** On repo 1 it took 43, 60, 40 turns (the harness's own count); it first wrote its report 5, 8, 3 minutes in and its last model call came at minute 5, 8, 3. For scale: Opus 5.5 at medium took 58-74 turns there over three runs of 5-6 minutes, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 69, 61, 141 turns (the harness's own count); it first wrote its report 7, 10, 18 minutes in and its last model call came at minute 7, 10, 19. For scale: Opus 5.5 at medium took 116-129 turns there over three runs of 11-12 minutes, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 676,796 tokens; 0 compactions across 6 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. 4 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 20, 22, 25, 27, 29, 30, 33, 34, 37, 39, 40, 43, 45, 46, 47, 50, 60, 61, 62, 68, 70, 71, 76, 77, 82, 83, 84, 91, 92, 95, 97, 98, 99, 100 ], "bug_hits": { "13": 1, "20": 3, "22": 1, "25": 2, "27": 1, "29": 2, "30": 3, "33": 1, "34": 1, "37": 1, "39": 3, "40": 1, "43": 3, "45": 3, "46": 1, "47": 2, "50": 3, "60": 2, "61": 1, "62": 1, "68": 1, "70": 1, "71": 3, "76": 3, "77": 3, "82": 1, "83": 3, "84": 2, "91": 1, "92": 3, "95": 1, "97": 3, "98": 3, "99": 3, "100": 3 }, "coverage_runs": 3, "turns": 138, "turns_repo1": 48, "turns_repo2": 90, "group_rank": 503 }, { "id": "Qwen3.8-Flash (low effort)", "n_runs": 1, "model": "Qwen3.8-Flash", "vendor": "Alibaba", "local": false, "harness": "Claude Code / Alibaba API", "effort": "low", "effort_status": "verified", "date": "2026-09-11", "color": "#F5D49B", "fixed": 23, "repo1_fixed": 11, "repo2_fixed": 12, "partial": 0, "claimed_only": 3, "extras": 7, "false_positive_fixes": 0, "wall_min": 98.1, "cost_usd": 1.37, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "One run at this tier. More importantly, read this row with its max companion rather than as the bottom rung of a working ladder: the same probe shows the dial SATURATES above low, with high, xhigh and max mutually indistinguishable (ranges 13933-27933, 12523-24778 and 13767-19431, means falling as the nominal tier rises). So the honest picture of this model's effort control is two settings, not five. Cost is a list-price ESTIMATE and not a bill - DashScope exposes no credits API on this path - computed from the run's own token counts at 0.15 in / 0.20 cache-write / 0.016 cache-read / 0.47 out per MTok, rates cross-checked against OpenRouter's Alibaba-endpoint listing, which matches Alibaba's published rates exactly on the Qwen3.8-Max row this board already carries.", "note": "The board's first Qwen row that does not go through an aggregator: Claude Code straight onto Alibaba's own Anthropic-compatible endpoint, no local proxy and no OpenRouter in the path. 23/105 for $1.37 at list puts it between the two DeepSeek V4.1 Flash rows (19/105 at $0.31 and 24/105 at $1.08) and above models costing twenty times more. The effort label here is MEASURED, not merely requested, which is unusual for a cheap model on this board: the endpoint 400-rejects an invented tier and names its accept-list (low, medium, high, xhigh, max), and a magnitude probe at n=3 per tier separates the bottom of the dial cleanly - mean output tokens run low=3557 against high=19329, low and medium sit in bands that do not overlap the top three, and the 5.43x spread between lowest and highest mean clears the worst within-tier spread of 2.00x. So low is a real setting on this path rather than a word in a column. The honesty profile is among the better ones here: 23 strict matches, ZERO partials, only 3 claimed-only across both repos, and 7 genuine unplanted extras.", "fixed_bugs": [ 13, 20, 22, 23, 25, 26, 27, 29, 30, 40, 43, 49, 50, 60, 71, 76, 83, 84, 91, 95, 97, 98, 99 ], "turns": 325, "turns_repo1": 136, "turns_repo2": 189, "group_rank": 304 }, { "id": "GPT-5.6 Luna (xhigh effort)", "n_runs": 1, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-08-28", "color": "#79C6AF", "fixed": 23, "repo1_fixed": 10, "repo2_fixed": 13, "partial": 2, "claimed_only": 0, "extras": 53, "false_positive_fixes": 0, "wall_min": 135.6, "cost_usd": 2.5, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 17, 20, 22, 25, 29, 30, 35, 40, 43, 46, 47, 49, 70, 71, 76, 77, 83, 84, 92, 98, 99, 102 ], "turns": 813, "turns_repo1": 464, "turns_repo2": 349, "group_rank": 601 }, { "id": "Opus 5.5 (low effort)", "n_runs": 1, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 23, "repo1_fixed": 11, "repo2_fixed": 12, "partial": 1, "claimed_only": 4, "extras": 14, "false_positive_fixes": 0, "wall_min": 12.4, "cost_usd": 8.68, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (24.0 of 105). This row reported 23 of 105 from a single run. The replicates are byte-identical re-runs; across 2 the runs span 23 to 25. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean - and this board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw. Two replicates are in flight and will supersede it. Runs a 1M context window; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. The wall figure is contended.", "note": "Anthropic's Opus 5.5 at LOW effort through Claude Code on the first-party API - a rung below medium, added because the four rungs above it were replicated three times each and the ladder resolved at the top and not at the bottom. 23 of 105: 11 of 45 on repo 1 and 12 of 60 on repo 2, 14 genuine extras, 12.4 minutes, $8.68. ONE RUN so far; two replicates are running and will supersede this row. *** WHY THIS RUNG EXISTS *** at three runs a rung, xhigh beats high by 4.3 points of 105 and max beats xhigh by 5.7, both wider than the spread measured inside either rung. But high beats medium by 1.4 against spreads of 1 and 1 - one bug of 105, between configurations whose own runs move by that much. Twelve runs did not separate the bottom two rungs. Going a rung lower is the direct test of whether the dial does anything at all down there, and it is a cheaper question to answer than it looks: low is the fastest and cheapest setting on the ladder. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** CONFIGURED BYTE-IDENTICAL TO THE RUNG ABOVE IT *** same model string claude-opus-5-5[1m], same 1M window, same timeout, same price table, same blind judge. One string differs, the effort flag. *** COST IS A LIST ESTIMATE *** at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** THE CONTEXT WINDOW *** this run holds a 1M window, so it compares cleanly to the rest of the Opus 5.5 ladder and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default. *** THE WALL FIGURE ON THIS RUNG IS CONTENDED AND THE OTHERS ARE NOT *** the four rungs above it were measured on an otherwise idle machine; this one ran alongside another vendor's legs, deliberately, to get the rung measured sooner. Contention can only INFLATE a wall, so a low wall here is still a real low wall - but do not read a minute-for-minute comparison against the rungs above. Token spend is the figure that shows whether the effort flag reached the model, and contention does not touch it. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it.", "fixed_bugs": [ 16, 20, 25, 26, 29, 30, 33, 34, 39, 40, 43, 47, 49, 50, 71, 76, 83, 84, 92, 98, 99, 100, 102 ], "turns": 137, "turns_repo1": 51, "turns_repo2": 86, "group_rank": 204 }, { "id": "Grok 4.6 (medium)", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "medium", "effort_status": "verified", "date": "2026-08-14", "color": "#757A82", "fixed": 23, "repo1_fixed": 7, "repo2_fixed": 16, "partial": 0, "claimed_only": 2, "extras": 11, "false_positive_fixes": 0, "wall_min": 20.5, "cost_usd": 13.35, "cost_kind": "floor", "wall_note": "Both repo legs ran concurrently on one machine and one key; contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": "Sequential re-run on 2026-08-28 (22/105) supersedes this concurrent-legs run for a comparable wall.", "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 30, 33, 34, 43, 45, 47, 49, 50, 60, 62, 71, 76, 77, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 103 }, { "id": "Grok 4.6 (high)", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "verified", "date": "2026-08-28", "color": "#757A82", "fixed": 23, "repo1_fixed": 7, "repo2_fixed": 16, "partial": 0, "claimed_only": 1, "extras": 16, "false_positive_fixes": 0, "wall_min": 33.0, "cost_usd": 15.73, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": "The missing Grok 4.6 dial tier (between medium 23 and xhigh, a rung since run three times for a mean of 29 - this row is still a single run and the gap above it is smaller than it looked when both endpoints were n=1), run on Windows when the macOS delegation stalled. Effort verified active by zero-token ACP readback. Wall ran alongside the Codex xhigh chains on a separate subscription - minor concurrency.", "fixed_bugs": [ 13, 16, 20, 30, 33, 34, 43, 47, 49, 50, 60, 62, 76, 77, 82, 83, 84, 92, 95, 98, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 102 }, { "id": "MiMo-V2.6-Pro - mean of 3", "n_runs": 3, "model": "MiMo-V2.6-Pro", "vendor": "Xiaomi", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-22", "color": "#1A1AA6", "fixed": 22.7, "repo1_fixed": 11.3, "repo2_fixed": 11.3, "partial": 1, "claimed_only": 2, "extras": 5.3, "false_positive_fixes": 0, "wall_min": 90.9, "cost_usd": 0.86, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 20, 24 and 24 of 105 - spread 4. The middle run was judged by Grok 4.7 rather than the Codex gpt-5.5 used for the other two - using a grok judge there was a deliberate per-run choice, though the 4.7 checkpoint was served against a request for 4.5 - but it means this mean blends two judge standards. Cost is a real OpenRouter bill across all three runs. Effort is `default`: this route's reasoning field is accepted but measured inert on this model, so this is not a measurement of the model at any particular reasoning level. Runs shared the laptop with benchmark legs from other vendors, so wall figures are contended and biased upward; scores and cost are unaffected.", "note": "Xiaomi's MiMo-V2.6-Pro driven by Claude Code through an Anthropic-API shim onto OpenRouter, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 20, 24 and 24 of 105 (repo 1: 9, 13, 12 of 45 - repo 2: 11, 11, 12 of 60). Mean 22.7, spread 4. *** THIS SUPERSEDES THE MEAN OF TWO (22.0 of 105) *** a third independent run tightens the estimate rather than changing its shape - the mean moved from 22.0 to 22.7. *** JUDGE IS NOT UNIFORM ACROSS THE THREE RUNS, BY CHOICE - SAID PLAINLY *** the first and third runs (mimo26pro, mimo26pror3) were judged by blind Codex gpt-5.5. The second run (mimo26pror2) was judged by Grok instead - a deliberate choice for this replicate, both being valid non-sibling judges for a Xiaomi model. The CHECKPOINT was not chosen: the run asked for grok-4.5 and the endpoint served grok-4.7, which the receipt records and this note originally did not. This mean therefore blends two judge standards rather than replicating one; read the spread with that in mind alongside ordinary run-to-run variance. *** WALL AND COST *** Walls 56.4, 79.4, 136.9 minutes, mean 90.9; cost $0.79, $0.77, $1.03, mean $0.86. COST IS A REAL BILL, the OpenRouter credits delta across each run's legs, comparable to other billed rows here and not to list-priced or floor-reconstructed ones. *** GENUINE EXTRAS *** 3, 5 and 8 across the three runs, mean 5.3 - real defects found in the repos that are not part of the planted set. *** THE DIAL, OR LACK OF ONE *** this route's `reasoning` field is accepted but measured inert on this model (see the sibling first-party and default-row notes on this board); `default` never named a state that could be asked for again, and none of the three runs in this mean tried to assert one. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run, one repo at a time. The runs shared the laptop with legs from other vendors, which the row's wall marker discloses; scores, tokens and cost are untouched by that - only the clock is.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 26, 27, 30, 32, 33, 34, 39, 40, 43, 45, 47, 49, 50, 60, 62, 70, 71, 76, 77, 79, 83, 84, 92, 99, 100 ], "bug_hits": { "13": 2, "16": 1, "20": 3, "21": 2, "22": 1, "23": 1, "25": 3, "26": 1, "27": 1, "30": 3, "32": 2, "33": 3, "34": 3, "39": 2, "40": 1, "43": 3, "45": 2, "47": 2, "49": 3, "50": 2, "60": 2, "62": 1, "70": 3, "71": 3, "76": 3, "77": 1, "79": 1, "83": 3, "84": 1, "92": 3, "99": 3, "100": 3 }, "coverage_runs": 3, "turns": 308, "turns_repo1": 129, "turns_repo2": 180, "group_rank": 6 }, { "id": "GPT-6.1 Sol (low effort) - mean of 2", "n_runs": 2, "model": "GPT-6.1 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-10-01", "color": "#2BA580", "fixed": 22.5, "repo1_fixed": 11.5, "repo2_fixed": 11, "partial": 3.5, "claimed_only": 0.5, "extras": 24, "false_positive_fixes": 0, "wall_min": 25.4, "cost_usd": 1.34, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Two runs of an identical configuration scoring 31 and 14 of 105 - spread 17. Read the mean with that width - two runs is a thin mean. It runs Codex 0.159.0, newer than every other OpenAI row, because the build the others use refuses this model. Cost is a list-rate estimate; no request reached the long-context rates. Wall figures may include load from other work on the laptop.", "note": "OpenAI's GPT-6.1 Sol at low effort through OpenAI's own Codex CLI, on a ChatGPT account, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 31 and 14 of 105 (repo 1: 15, 8 - repo 2: 16, 6). Mean 22.5, range 17. *** THIS SUPERSEDES GPT-6.1 SOL (LOW EFFORT) (31 OF 105) *** *** THE REPLICATE IS BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, pinned CLI, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. *** WALL, COST AND TURNS *** walls 36.5, 14.1 minutes; cost $1.77, $0.90; turns 91, 57. *** GENUINE EXTRAS *** 32, 16 across the runs, never folded into the score. The GPT-6.1 Sol ladder elsewhere on the board: GPT-6.1 Sol at medium 29 (mean of 2), GPT-6.1 Sol at high 36.5 (mean of 2), GPT-6.1 Sol at xhigh 42.7 (mean of 3), GPT-6.1 Sol at max 44.3 (mean of 3). At the same tier the board publishes GPT-6 Sol at low 6, GPT-6 Astra at low 27, Sonnet 5.5 at low 19 (mean of 3) of 105. *** THIS ROW RUNS A NEWER CODEX BUILD THAN THE OTHER OPENAI ROWS, UNAVOIDABLY *** the build the rest of this board's OpenAI rows use answers 'not supported when using Codex with a ChatGPT account' for this model on the same account at the same minute that Codex 0.159.0 runs it, so the blocker is the client, not entitlement. The GPT-6 Sol rows ran on 0.155.1, so a gap against them is model-plus-client. (That rejection message is generic - a made-up slug gets it word for word - so it is never evidence about whether a model exists; the API's own model list settled that.) *** THE TIER IS VERIFIED, NOT ASSUMED *** the effort is a first-party Codex tier from the account's own model list, and every leg's own Codex session record shows it ran at low. *** COST IS A LIST ESTIMATE *** token counts at OpenAI's published GPT-6.1 Sol short-context rates, $2 input, $0.10 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output; none of these legs sent one - Codex caps this model's context below that line, and the session records confirm every request stayed under it. The Codex CLI on a subscription exposes no per-run bill, hence `list`. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt. *** RECEIPTS *** Every leg exited 0 with a report written, run SEQUENTIALLY, one repo at a time, code mode off as on the GPT-6 Sol arms. No other benchmark leg overlapped these legs - the only load this board's marker records - but the laptop also ran interactive coding-agent sessions over these nights, at times near full CPU, so wall figures may run high; scores, tokens and cost are unaffected.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 37, 40, 41, 43, 44, 45, 47, 49, 50, 62, 68, 69, 76, 77, 82, 83, 84, 92, 98, 99, 100, 102 ], "bug_hits": { "13": 2, "16": 2, "20": 2, "22": 2, "25": 1, "29": 1, "30": 2, "33": 2, "34": 2, "37": 1, "40": 1, "41": 1, "43": 1, "44": 1, "45": 2, "47": 1, "49": 2, "50": 2, "62": 1, "68": 2, "69": 1, "76": 1, "77": 1, "82": 1, "83": 2, "84": 2, "92": 1, "98": 1, "99": 1, "100": 2, "102": 1 }, "coverage_runs": 2, "turns": 74, "turns_repo1": 40, "turns_repo2": 34, "group_rank": 104 }, { "id": "Opus 5.5 (low effort) - mean of 3", "n_runs": 3, "model": "Opus 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-23", "color": "#DE6C48", "fixed": 22.3, "repo1_fixed": 11.3, "repo2_fixed": 11, "partial": 0.7, "claimed_only": 2.4, "extras": 9.7, "false_positive_fixes": 0, "wall_min": 11.5, "cost_usd": 8.34, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 23, 25 and 19 of 105 - spread 6. Read the mean with that width, not as a point. Runs a 1M context window with zero compactions; every Opus 5 row here runs 200K and compacted, so this ladder does not compare cleanly to that one. Cost is a list-rate estimate, not an invoice. Wall figures are contended.", "note": "Anthropic's Opus 5.5 at low effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 23, 25 and 19 of 105 (repo 1: 11, 13, 10 of 45 - repo 2: 12, 12, 9 of 60). Mean 22.3, spread 6. *** THIS SUPERSEDES OPUS 5.5 (LOW EFFORT) - MEAN OF 2 (24.0 OF 105) *** *** WHY THE REPLICATES EXIST *** the first pass measured this ladder at one run per rung and it came out monotone - medium 30, high 32, xhigh 35, max 43. That was suggestive and nothing more: no gap in it exceeded the 13-point spread this board has measured between two runs of ONE identical max-effort configuration, so the ordering of any two adjacent rungs was unresolved and every note said so. Three runs per rung is what can resolve it, and the whole ladder is being re-measured that way rather than only its top. The board currently reads low 22.3 (n=3), medium 30.3 (n=3), high 31.7 (n=3), xhigh 36 (n=3), max 41.7 (n=3). *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string claude-opus-5-5[1m], same effort, same timeout, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** WALL AND COST *** Walls 12.4, 10.2, 12.0 minutes, mean 11.5; cost $8.68, $8.79, $7.55, mean $8.34. COST IS A LIST ESTIMATE at Anthropic's published Opus 5.5 rates - $4 input, $20 output, $0.20 cache read (a 0.05x cache multiplier unique to this model, where every other model here uses 0.1x) and $8 cache write, the 1-hour rate, which is what this harness actually writes and what the streams confirm. Not an invoice, hence `list`. *** GENUINE EXTRAS *** 14, 5 and 10 across the runs, mean 9.7 - real defects found in the repos that nobody planted, reported separately and never folded into the score. *** THE CONTEXT WINDOW *** every run here holds a 1M window and crosses zero compaction boundaries, so these rows compare cleanly to each other and NOT to the Opus 5 ladder, which ran the CLI's default 200K and compacted repeatedly mid-task. Autocompact ran at the CLI's own default throughout. *** RECEIPTS *** All legs exited 0, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 on every leg - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The runs shared the laptop with legs from other vendors, which the wall marker discloses; scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 16, 20, 22, 23, 25, 26, 29, 30, 33, 34, 39, 40, 43, 44, 45, 47, 49, 50, 60, 62, 71, 76, 77, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 3, "20": 3, "22": 2, "23": 1, "25": 2, "26": 1, "29": 3, "30": 3, "33": 3, "34": 3, "39": 3, "40": 1, "43": 3, "44": 1, "45": 1, "47": 3, "49": 3, "50": 1, "60": 1, "62": 2, "71": 2, "76": 3, "77": 1, "83": 2, "84": 3, "92": 1, "97": 1, "98": 2, "99": 3, "100": 3, "102": 2 }, "coverage_runs": 3, "turns": 127, "turns_repo1": 55, "turns_repo2": 71, "group_rank": 204 }, { "id": "MiMo-V2.6-Pro - mean of 2", "n_runs": 2, "model": "MiMo-V2.6-Pro", "vendor": "Xiaomi", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-22", "color": "#1A1AA6", "fixed": 22, "repo1_fixed": 10.5, "repo2_fixed": 11.5, "partial": 0.5, "claimed_only": 2.5, "extras": 5.5, "false_positive_fixes": 0, "wall_min": 96.6, "cost_usd": 0.91, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-22 by the mean of three (22.7 of 105). This row was the mean of two runs at 22.0, and it was featured. A third independent run tightened the estimate to 20, 24 and 24 across the three. Nothing here is retracted - the n=2 mean is kept as the receipt for what the board showed before this third run.", "caveat": "Two runs of an identical configuration scoring 20 and 24 of 105 - spread 4, in line with what this board has already measured elsewhere. Cost is a real OpenRouter bill across both runs. Effort is `default`: this route's reasoning field is accepted but measured inert on this model, so this is not a measurement of the model at any particular reasoning level. Runs shared the laptop with benchmark legs from other vendors, so wall figures are contended and biased upward; scores and cost are unaffected.", "note": "Xiaomi's MiMo-V2.6-Pro driven by Claude Code through an Anthropic-API shim onto OpenRouter, published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 20 and 24 of 105 (repo 1: 9, 12 of 45 - repo 2: 11, 12 of 60). Mean 22.0, spread 4. *** THIS SUPERSEDES A SINGLE RUN OF 20 *** the earlier row was one draw on a model nothing on this board had seen before; two runs of the same configuration give a spread instead of a single, unreplicated number. *** WALL AND COST *** Walls 56.4, 136.9 minutes, mean 96.7; cost $0.79, $1.03, mean $0.91. COST IS A REAL BILL, the OpenRouter credits delta across both runs' legs, comparable to other billed rows here and not to list-priced or floor-reconstructed ones. *** GENUINE EXTRAS *** 3 and 8 across the two runs, mean 5.5 - real defects found in the repos that are not part of the planted set. *** THE DIAL, OR LACK OF ONE *** this route's `reasoning` field is accepted but measured inert on this model (see the sibling first-party and default-row notes on this board); `default` never named a state that could be asked for again, and neither run in this mean tried to assert one. *** RECEIPTS *** All four legs exited 0. Legs SEQUENTIAL within every run, one repo at a time. The runs shared the laptop with legs from other vendors, which the row's wall marker discloses; scores, tokens and cost are untouched by that - only the clock is. Judge: blind Codex gpt-5.5, neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 30, 32, 33, 34, 39, 43, 45, 47, 49, 50, 60, 62, 70, 71, 76, 83, 84, 92, 99, 100 ], "bug_hits": { "13": 1, "16": 1, "20": 2, "21": 1, "22": 1, "23": 1, "25": 2, "30": 2, "32": 1, "33": 2, "34": 2, "39": 2, "43": 2, "45": 1, "47": 1, "49": 2, "50": 2, "60": 2, "62": 1, "70": 2, "71": 2, "76": 2, "83": 2, "84": 1, "92": 2, "99": 2, "100": 2 }, "coverage_runs": 2, "turns": 326, "turns_repo1": 131, "turns_repo2": 195, "group_rank": 6 }, { "id": "Grok 4.6 (medium) seq", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "medium", "effort_status": "verified", "date": "2026-08-28", "color": "#757A82", "fixed": 22, "repo1_fixed": 9, "repo2_fixed": 13, "partial": 0, "claimed_only": 2, "extras": 11, "false_positive_fixes": 0, "wall_min": 26.5, "cost_usd": 5.8, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": "Sequential re-run of the 08-14 concurrent-legs medium (23/105 vs this 22/105 - n=2 agreement); comparable single-machine wall, so it replaces the dotted original.", "fixed_bugs": [ 13, 16, 20, 30, 33, 34, 39, 43, 45, 47, 49, 50, 60, 62, 76, 83, 84, 92, 98, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 103 }, { "id": "Haiku 5.5 (max effort)", "n_runs": 1, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-10-07", "color": "#EBA691", "fixed": 22, "repo1_fixed": 10, "repo2_fixed": 12, "partial": 2, "claimed_only": 6, "extras": 13, "false_positive_fixes": 0, "wall_min": 34.9, "cost_usd": 6.92, "cost_kind": "list", "wall_note": "Both repo legs ran at the same time on one account; contention can inflate this figure, not shrink it.", "ceiling": true, "featured": false, "superseded": "Superseded 2026-10-07 by the mean of two runs (21.5 of 105): this row was run 1 alone, at 22; run 2 scored 21. Nothing here is retracted - it is the receipt for what the board showed before run 2.", "caveat": "One run at max effort, both repo legs at once on one account. Single runs on this board move by more than 2 points against a three-run mean about half the time. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at max effort through Claude Code on the first-party API, released 2026-10-07. ONE RUN: 22 of 105 - 10 of 45 on repo 1 and 12 of 60 on repo 2 - with 2 partial fix(es), 6 claimed-only and 13 genuine unplanted extras; 34.9 minutes, $6.92. The same model's xhigh row is a mean of three runs; this is a single run, and on this board a first run has moved by more than 2 points against its own three-run mean about half the time. *** EFFORT: MAX, THE CEILING, READ OFF THE WIRE *** Claude Code 2.1.292 does not know this model yet, so the request was captured: the main loop sends output_config.effort=max with adaptive thinking to claude-haiku-5-5, and the API accepts it. A first-party tier, accepted, not magnitude-probed. Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** BOTH REPO LEGS RAN AT THE SAME TIME *** on one account, so each leg's wall includes the other's contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and each leg's priced responses were checked to be exactly the ones in its own stream, with none shared between the legs. *** COST, PRICED PER REQUEST *** 203 of 218 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates; 98.6% of prompt tokens were cache reads. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 20, 22, 25, 26, 27, 28, 29, 30, 43, 45, 47, 49, 50, 71, 76, 77, 82, 84, 97, 98, 99, 100 ], "turns": 371, "turns_repo1": 137, "turns_repo2": 234, "group_rank": 700 }, { "id": "Gemini 3.7 Flash (high)", "n_runs": 1, "model": "Gemini 3.7 Flash", "vendor": "Google", "local": false, "harness": "Gemini CLI (retired) + model-pinning gateway", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-08-24", "color": "#6697E9", "fixed": 22, "repo1_fixed": 8, "repo2_fixed": 14, "partial": 0, "claimed_only": 0, "extras": 4, "false_positive_fixes": 0, "wall_min": 96.8, "cost_usd": 8.43, "cost_kind": "list", "wall_note": "Both repo legs ran concurrently, AND this is the one adjusted figure on the board: a 35-minute harness stall is excluded.", "ceiling": false, "featured": false, "superseded": null, "caveat": "Gemini CLI was retired on Jun 18, 2026; it silently served 3.5 Flash until the model was pinned on the wire. Wall clock excludes a 35-minute harness stall.", "note": null, "fixed_bugs": [ 16, 20, 28, 30, 33, 34, 43, 45, 47, 50, 51, 60, 62, 66, 69, 71, 76, 82, 84, 97, 99, 100 ], "turns": 530, "turns_repo1": 254, "turns_repo2": 276, "group_rank": 202 }, { "id": "DeepSeek V4.1 Flash (max effort) - mean of 3", "n_runs": 3, "model": "DeepSeek V4.1 Flash", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / DeepSeek API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-15", "color": "#5A3EC8", "fixed": 21.7, "repo1_fixed": 11.3, "repo2_fixed": 10.3, "partial": 0.7, "claimed_only": 1, "extras": 6.7, "false_positive_fixes": 0, "wall_min": 55.9, "cost_usd": 0.78, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 19 to 24. This row SUPERSEDES the single-run max row (24/105) published 2026-09-10, which was the HIGHEST of the three; the supersession is about method, not a correction, and that receipt stands. Every figure here is a MEAN carried to one decimal - 21.7 of 105. The three runs were verified to have run in the same harness regime on both repos, which is the exception on this board rather than the rule; the CLI version differed (2.1.267 for the first run, 2.1.270 for the other two).", "note": "DeepSeek's top Flash rung, RUN THREE TIMES. Runs scored 24, 19 and 22 of 105 (repo 1: 14, 10, 10 - repo 2: 10, 9, 12). Mean 21.7, range 5. *** THE ONE ARM IN THIS WAVE WHOSE REPLICATION IS VERIFIED *** Three flash arms were replicated on the same nights; this is the only one where the harness left enough behind to check that the three runs ran in the same configuration. All six legs resolved deepseek-flash with the 1M window, and NONE of them compacted: peak live context 513,330 / 509,936 / 471,821 on repo 1 and 520,206 / 544,170 / 558,074 on repo 2, against a harness that would have thrown context away at about 30 per cent had it been triggered. A mean is a claim that its members differ only by chance, and on this row that claim was tested rather than assumed. *** ONE THING WAS NOT HELD FIXED *** The 2026-09-10 run ran Claude Code 2.1.267 and both replicates ran 2.1.270. The CLI self-updates between waves. It is stated rather than buried because the resolved model and the compaction behaviour are identical across all six legs, so the version string is the only difference the receipts show. *** WHAT THE SPREAD LOOKS LIKE *** The totals span 5 points, and the two repos moved in OPPOSITE directions: repo 1 fell 14, 10, 10 while repo 2 rose 10, 9, 12. That is an observation about three runs, not a mechanism, and it is not a claim that totals are steadier than legs - three draws cannot establish that. The published single run scored 24, the TOP of its own three-run range, so every comparison drawn against 24 was drawn against this configuration's best day. *** THE WALL COLUMN ON THIS ROW IS NOISY AND THE PER-LEG FIGURES ARE THE HONEST FORM *** Against the control's tokens per minute the four replicate legs ran 47, 131, 118 and 104 per cent - in both directions, same binary, same night. The 47 per cent leg is the one that overlapped hardest with two other arms running concurrently. Do not read a ratio off this row. *** EVER VS ALWAYS - THE THING A SINGLE RUN CANNOT SHOW *** Across the three runs this arm ever fixed 33 distinct bugs of 105, well above the 21.7 any single run manages. Of those 33, only 11 fell in ALL THREE runs; 10 fell in two and 12 in exactly one. A third of what it ever fixed is what it reliably fixes. It is the LEAST consistent of the three flash arms replicated in this wave despite scoring the highest of them, which is exactly the trade a single number hides. The Coverage view shades each tick by its observed hit rate, so this is readable on the row. *** RECEIPTS *** All six legs exited 0. Per-leg (repo 1 / repo 2): wall 25.4/17.2, 59.9/18.1 and 20.6/26.4 minutes; bill 0.62/0.46, 0.30/0.29 and 0.31/0.36 dollars. Genuine extras 7, 6 and 7. Claimed-only 0, 2 and 1 across six legs, all on repo 2. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. billing=REAL bill: the runner snapshots the DeepSeek balance before and after each leg and writes the drop, so this cost column is money that moved, not a list estimate. Judge: Codex gpt-5.5 on all six legs.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 27, 28, 29, 30, 33, 34, 39, 41, 43, 44, 45, 47, 49, 50, 57, 60, 70, 71, 76, 83, 84, 92, 97, 98, 99, 100 ], "bug_hits": { "13": 2, "16": 1, "20": 2, "21": 1, "22": 1, "23": 2, "25": 2, "27": 3, "28": 2, "29": 3, "30": 3, "33": 2, "34": 1, "39": 2, "41": 1, "43": 3, "44": 1, "45": 2, "47": 1, "49": 1, "50": 3, "57": 1, "60": 1, "70": 3, "71": 2, "76": 3, "83": 3, "84": 1, "92": 3, "97": 2, "98": 3, "99": 3, "100": 1 }, "coverage_runs": 3, "turns": 461, "turns_repo1": 193, "turns_repo2": 268, "group_rank": 0 }, { "id": "Haiku 5.5 (max effort) - mean of 2", "n_runs": 2, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-10-07", "color": "#EBA691", "fixed": 21.5, "repo1_fixed": 10, "repo2_fixed": 11.5, "partial": 1, "claimed_only": 3.5, "extras": 9, "false_positive_fixes": 0, "wall_min": 32.5, "cost_usd": 5.83, "cost_kind": "list", "wall_note": "Every run took its two repo legs at the same time on one account; contention can inflate this figure, not shrink it.", "ceiling": true, "featured": true, "superseded": null, "caveat": "Mean of two runs of one configuration (22 and 21 of 105), each with both repo legs at once on one account. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at max effort through Claude Code on the first-party API, released 2026-10-07. MEAN OF TWO RUNS of one configuration: 22 and 21 of 105 (repo 1: 10 and 10 of 45 - repo 2: 12 and 11 of 60), mean 21.5. Claimed-only fixes 6 and 1; genuine unplanted extras 13 and 5. Walls 34.9 and 30.3 minutes; cost $6.92 and $4.72. Effort max is this model's top tier and was read off the wire (main loop sends output_config.effort=max with adaptive thinking; the API accepts it). A first-party Claude Code tier, accepted, not magnitude-probed; Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** EVERY RUN TOOK ITS TWO REPO LEGS AT THE SAME TIME *** on one account, so walls include contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and in every run each leg's priced responses were checked to be exactly the ones in its own stream, with none shared between the legs. *** COST, PRICED PER REQUEST *** 345 of 371 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 20, 21, 22, 25, 26, 27, 28, 29, 30, 39, 43, 45, 47, 49, 50, 62, 71, 76, 77, 82, 84, 97, 98, 99, 100 ], "bug_hits": { "20": 2, "21": 1, "22": 2, "25": 2, "26": 1, "27": 2, "28": 1, "29": 2, "30": 2, "39": 1, "43": 2, "45": 2, "47": 2, "49": 2, "50": 1, "62": 1, "71": 2, "76": 2, "77": 2, "82": 1, "84": 2, "97": 2, "98": 2, "99": 2, "100": 2 }, "coverage_runs": 2, "turns": 328, "turns_repo1": 120, "turns_repo2": 208, "group_rank": 700 }, { "id": "Fable 5.1 (medium effort)", "n_runs": 1, "model": "Fable 5.1", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-10", "color": "#D9542B", "fixed": 21, "repo1_fixed": 8, "repo2_fixed": 13, "partial": 2, "claimed_only": 3, "extras": 3, "false_positive_fixes": 0, "wall_min": 23.0, "cost_usd": 23.73, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "One run per tier. A single non-monotonic point is a finding about this run, not a demonstrated property of the medium tier - the board's own variance evidence (three Grok 4.5 runs at one setting scoring 16, 13 and 17) is the reason that distinction is enforced here. What the row does establish is that the shell-first strategy and the low score arrived together, which is a mechanism worth having on record either way.", "note": "Fable 5.1's dial does not run in a straight line: medium scores 21/105, BELOW its own low row (29) and well below high (33) and max (43). The run is clean - both legs exited success, no truncation, no permission denials, no tool errors - and the raw captures say why in the tool mix rather than in the thinking. At medium the model worked almost entirely through the shell: 83 Bash calls against 3 Reads and ZERO Edits or Greps on repo 1, and 99 Bash against 2 Reads and zero Edits on repo 2. At low it read the codebase instead - 44 Reads, 20 Greps, 16 Edits on repo 1. That shows up in the billing shape too: medium pulled 4.0M cached-read tokens on repo 1 against low's 19.7M, because a session that never reads files into context keeps a small context to re-send. Fewer bugs found is what you would expect from a model that barely looked at the code.", "fixed_bugs": [ 13, 16, 20, 30, 36, 39, 43, 45, 47, 60, 70, 71, 76, 80, 82, 83, 84, 92, 99, 100, 102 ], "turns": 194, "turns_repo1": 88, "turns_repo2": 106, "group_rank": 3 }, { "id": "Kimi K3", "n_runs": 1, "model": "Kimi K3", "vendor": "Moonshot AI", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-07-26", "color": "#9333EA", "fixed": 21, "repo1_fixed": 4, "repo2_fixed": 17, "partial": 0, "claimed_only": 6, "extras": 6, "false_positive_fixes": 0, "wall_min": 107.8, "cost_usd": 25.27, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 20, 25, 30, 43, 49, 50, 51, 60, 62, 69, 71, 76, 77, 83, 84, 91, 92, 97, 98, 99, 102 ], "turns": 394, "turns_repo1": 138, "turns_repo2": 256, "group_rank": 6 }, { "id": "Opus 5", "n_runs": 1, "model": "Opus 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-07-26", "color": "#E17757", "fixed": 21, "repo1_fixed": 11, "repo2_fixed": 10, "partial": 0, "claimed_only": 3, "extras": 6, "false_positive_fixes": 0, "wall_min": 37.4, "cost_usd": 41.51, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 21, 22, 27, 30, 33, 34, 39, 43, 45, 47, 50, 60, 70, 71, 76, 83, 92, 98, 99 ], "turns": 261, "turns_repo1": 121, "turns_repo2": 140, "group_rank": 302 }, { "id": "Muse Spark 1.3 (xhigh effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "xhigh", "effort_status": "verified", "date": "2026-09-14", "color": "#E57EA2", "fixed": 20.3, "repo1_fixed": 8, "repo2_fixed": 12.3, "partial": 0.7, "claimed_only": 0, "extras": 13.3, "false_positive_fixes": 0, "wall_min": 63.8, "cost_usd": 13.71, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 16 to 25. This row SUPERSEDES the single-run xhigh row (20/105) published 2026-09-13, which landed within a point of the mean - the supersession is about method, not a correction. Every figure on this row is a MEAN of the three runs, carried to one decimal - 20.3 of 105. max has since been run three times too (33, 33, 35; mean 33.7, range 2), so nothing on this dial is n=1 any more. The step from xhigh's mean to max is 13.4 points against ranges of 9 here and 2 there - the one gap on this dial that run-to-run noise does not explain.", "note": "The FOURTH rung of Meta's first-party dial to be run three times, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 20, 25 and 16 of 105 (repo 1: 8, 10, 6 - repo 2: 12, 15, 10). Mean 20.3, range 9. *** WHERE THIS LEAVES THE DIAL *** ALL FIVE rungs have since been replicated: 9.7, 13.0, 18.7, 20.3 and 33.7. The dial is monotone across all five, and max was the last to be run three times. The step from high's mean to xhigh's is 1.7 points, against three-run ranges of 3 at high and 9 at xhigh, so these two rungs are not separated by anything this board can measure - which is what the single-run pair (19 and 20) already suggested and is now on firmer ground. *** A CLAIM THIS ROW DOES NOT MAKE, AND WHY IT IS WORTH SAYING *** The three-run ranges came out 9, 6, 3 and now 9. An earlier version of the low, medium and high rows read the first three as a spread that narrows as effort rises, and generalised it to every row on this board. That was withdrawn on 2026-09-14 BEFORE this rung was scored, because a range from three draws is a very noisy estimate of spread and three of them cannot establish a trend in variance at all. This rung landing at 9 is consistent with the withdrawal but is not the reason for it: a fourth number could not have settled a question that needs n in the tens per rung. The general rule this board now applies: a shape across a handful of summary statistics is an observation, and an observation is not a finding. *** WHAT IS SOLID HERE *** Two things, both counts rather than inferences. A fixed configuration - same model, same prompt, same harness, same route, same effort flag, same judge - produced totals of 16, 20 and 25 on this rung, a 9-point range on a 105-bug benchmark. And across its three runs this rung ever fixed 30 distinct bugs. Nothing about the distribution of repeats is claimed; those are counts of 8, 13 and 9 bugs and they move on very little. All six legs were clean: the Linux install returned 0, the shim logged zero errors, every leg exited 0, and no leg carries a handicap note. Per-leg receipts (repo 1 / repo 2): calls 417/206, 294/330, 242/325; wall 73.0, 61.7 and 56.5 minutes; cost 15.58, 13.17 and 12.38 dollars. ZERO claimed-only across all six legs. Genuine extras 7, 24 and 9. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR: Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control. billing=LIST ESTIMATE, NOT A BILL: a subscription account with no per-run credits delta, costed at the same Meta list rates as the other first-party rows, which is what keeps them comparable. ctx 1M.", "fixed_bugs": [ 16, 20, 22, 25, 29, 30, 33, 34, 39, 43, 44, 45, 47, 49, 50, 51, 60, 62, 69, 71, 76, 77, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "16": 3, "20": 3, "22": 1, "25": 3, "29": 1, "30": 3, "33": 2, "34": 2, "39": 2, "43": 1, "44": 1, "45": 2, "47": 2, "49": 2, "50": 3, "51": 1, "60": 2, "62": 1, "69": 1, "71": 3, "76": 3, "77": 2, "83": 3, "84": 2, "92": 2, "97": 2, "98": 2, "99": 3, "100": 2, "102": 1 }, "coverage_runs": 3, "turns": 332, "turns_repo1": 132, "turns_repo2": 200, "group_rank": 201 }, { "id": "MiMo-V2.6-Pro", "n_runs": 1, "model": "MiMo-V2.6-Pro", "vendor": "Xiaomi", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "default", "date": "2026-09-22", "color": "#1A1AA6", "fixed": 20, "repo1_fixed": 9, "repo2_fixed": 11, "partial": 0, "claimed_only": 3, "extras": 3, "false_positive_fixes": 0, "wall_min": 56.4, "cost_usd": 0.79, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-22 by the first-party row (18/105, effort explicitly asserted). This row ran at `default` over an aggregator hop - the aggregator's reasoning dial is accepted but measured inert, so `default` never named a state that could be asked for again. The first-party row asserts the model's one real toggle (`thinking.type=enabled`) explicitly and drops the aggregator hop entirely; it scored 2 points lower (18 vs 20), inside the noise this board has already measured elsewhere (widest observed spread: 6 of 105) and expected here since both rows almost certainly ran with reasoning already on. Nothing here is retracted - the 20 happened and this is its receipt - but it is no longer the row that best answers whether reasoning is on for this model. The OpenRouter route itself was also independently replicated: see 'MiMo-V2.6-Pro - mean of 2' (mean of two runs, 22.0 of 105) for the same-route, same-configuration successor. This row is superseded on two separate axes - route (the first-party row) and sample size (the mean row) - neither displacing the other.", "caveat": "ONE RUN, not a mean. Models on this board have moved several points between runs of an identical configuration - the widest observed spread is 6 of 105 - so read this number with that much slack in either direction. Effort is `default`: the route's `reasoning` field is accepted but measured inert (n=3 per condition, 2026-09-22), so a dial does exist here, contrary to what an earlier version of this row said, but it does not change behaviour - this is still not a measurement of the model at any particular reasoning level. The wall figure is contended: other vendors' legs shared the machine.", "note": "Xiaomi's MiMo-V2.6-Pro driven by Claude Code through an Anthropic-API shim onto OpenRouter. 20 of 105: 9 of 45 on repo 1 and 11 of 60 on repo 2, 3 genuine extras, 56.4 minutes, $0.79. ONE RUN - a first measurement of a model nothing on this board has seen, published as a single run and labelled as one. *** CORRECTED 2026-09-22: THE DIAL EXISTS, IT IS JUST INERT *** This endpoint's supported_parameters do include `reasoning` (frequency_penalty, include_reasoning, max_tokens, presence_penalty, reasoning, response_format, stop, structured_outputs, temperature, tool_choice, tools and top_p) - an earlier version of this note read that list as proof no dial exists at all, which overstated it. `reasoning` is exactly the field OpenRouter's {effort: low|medium|high} convention uses, and this endpoint accepts it. A direct probe against the same pinned host (n=3 per condition, same prompt, no-field vs effort=low vs effort=high) found reasoning-token counts that fully overlap: 738-1704 no-field, 739-997 low, 814-1023 high, no ordering between low and high. Same failure mode already measured on Qwen3.8-Max on this board: accepted, not binding. The row is `default` because that is what ran, before this probe existed - there was a dial to test, and it does nothing. *** SINGLE FIRST-PARTY HOST, PINNED ANYWAY *** OpenRouter's endpoint API lists one host for this slug: Xiaomi's own, tag xiaomi/fp8, ctx 1,048,576 and max output 131,072. The provider pin with allow_fallbacks=false is set regardless, because a slug served by one host at launch is not a slug served by one host for the life of a run, and an unpinned row that silently becomes mixed-path averages hosts instead of measuring a model. The CLI model string carries [1m] because the upstream context is 1M and that suffix is the only context hint the CLI accepts - without it the CLI assumes 200K and thrashes on autocompact. *** COST IS A REAL BILL *** $0.79 is the OpenRouter credits delta across both legs, not a token estimate, which makes it directly comparable to the other billed rows here and not to the list-priced or floor-reconstructed ones. List rates for context: 0.435 in / 0.0036 cache-read / 0.87 out per MTok. *** RECEIPTS *** Both legs exited 0, run SEQUENTIALLY, one repo at a time. The run shared the laptop with benchmark legs from other vendors, which the wall marker on this row discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 16, 20, 22, 25, 30, 33, 34, 39, 43, 49, 50, 60, 62, 70, 71, 76, 83, 92, 99, 100 ], "turns": 329, "turns_repo1": 107, "turns_repo2": 222, "group_rank": 6 }, { "id": "GPT-6 Sol (high effort)", "n_runs": 1, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 20, "repo1_fixed": 10, "repo2_fixed": 10, "partial": 1, "claimed_only": 2, "extras": 15, "false_positive_fixes": 0, "wall_min": 33.9, "cost_usd": 4.16, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; it does compare cleanly to the other GPT-6 Sol rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs, so the wall figure is not contended.", "note": "OpenAI's GPT-6 Sol at high effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 20 of 105: 10 of 45 on repo 1 and 10 of 60 on repo 2, 15 genuine extras, 33.9 minutes, $4.16. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Sol ladder currently reads low 6, medium 14, high 20, xhigh 25, max 29.3 (n=3). For comparison the board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max and 43 at xhigh, and GPT-5.6 Sol at 43.5 (mean of 2) at max. This model's max rung replicated to three runs and landed below all of them, which is why the rungs beneath it are being measured at all: a sweep separates a weak model from a mis-set dial. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration, and this model's own max rung spans three runs. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-sol, same pinned executable, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows. The GPT-6 Sol rungs DO compare cleanly to each other - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for both legs, so these wall figures compare directly against other uncontended rows. (Corrected 2026-09-23: this row first published saying the run shared the laptop, which its receipts do not support - neither leg recorded a peer. The claim was a literal copied from a publish script, and the wall column is the evidence that the effort dial reaches the model at all, so it is worth getting right in both directions.)", "fixed_bugs": [ 16, 17, 20, 25, 29, 30, 33, 34, 39, 45, 47, 49, 50, 61, 77, 82, 83, 84, 99, 100 ], "turns": 162, "turns_repo1": 80, "turns_repo2": 82, "group_rank": 202 }, { "id": "Sonnet 5.5 (low effort)", "n_runs": 1, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 20, "repo1_fixed": 7, "repo2_fixed": 13, "partial": 1, "claimed_only": 3, "extras": 5, "false_positive_fixes": 0, "wall_min": 7.4, "cost_usd": 5.75, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 3 (19.0 of 105). This row reported 20 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 17 to 20. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat this as one draw. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at low effort through Claude Code on the first-party API, measured the evening it shipped. 20 of 105: 7 of 45 on repo 1 and 13 of 60 on repo 2, 5 genuine extras, 7.4 minutes, $5.75. ONE RUN. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** HOW IT SPENT THE RUN *** On repo 1 it took 52 turns (the harness's own count); it first wrote its report 3 minutes in and its last model call came at minute 3. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 67 turns (the harness's own count); it first wrote its report 4 minutes in and its last model call came at minute 4. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 355,093 tokens; 0 compactions across 2 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 20, 25, 30, 39, 43, 45, 46, 49, 50, 60, 71, 76, 77, 83, 84, 92, 98, 99, 100 ], "turns": 119, "turns_repo1": 52, "turns_repo2": 67, "group_rank": 504 }, { "id": "GPT-5.6 Terra (xhigh effort)", "n_runs": 1, "model": "GPT-5.6 Terra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-08-28", "color": "#69BFA6", "fixed": 20, "repo1_fixed": 9, "repo2_fixed": 11, "partial": 2, "claimed_only": 0, "extras": 29, "false_positive_fixes": 0, "wall_min": 56.6, "cost_usd": 8.98, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 25, 30, 33, 34, 35, 43, 44, 47, 62, 71, 76, 77, 83, 84, 97, 98, 99, 102 ], "turns": 259, "turns_repo1": 127, "turns_repo2": 132, "group_rank": 501 }, { "id": "Gemini 3.8 Flash", "n_runs": 1, "model": "Gemini 3.8 Flash", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-09-02", "color": "#4983E4", "fixed": 20, "repo1_fixed": 7, "repo2_fixed": 13, "partial": 1, "claimed_only": 1, "extras": 6, "false_positive_fixes": 0, "wall_min": 29.8, "cost_usd": 9.78, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row (18.0/105). This row is ONE run of three and it is the HIGHEST of the three: identical runs scored 20, 19 and 15. Nothing in it is retracted; it was the top of its own range. The replication could not be VERIFIED - the Antigravity CLI writes no session init record, so neither the context regime nor the harness version is recoverable for any run, including this one.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 30, 33, 34, 45, 47, 60, 62, 76, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "turns": 546, "turns_repo1": 262, "turns_repo2": 284, "group_rank": 102 }, { "id": "Muse Spark 1.3 (xhigh effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "xhigh", "effort_status": "verified", "date": "2026-09-13", "color": "#E57EA2", "fixed": 20, "repo1_fixed": 8, "repo2_fixed": 12, "partial": 0, "claimed_only": 0, "extras": 7, "false_positive_fixes": 0, "wall_min": 73.0, "cost_usd": 15.58, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same rung (20.3, rounding to 20). This row is one of three runs that scored 20, 25 and 16; it is kept because a single run is no longer how this rung is reported, and nothing in it is corrected by arithmetic. Read the mean row's range - 9 points - before quoting anything this row says about a 1-point gap to high.", "caveat": "One run at this tier, as at each of the others, so the curve is three points of n=1. What it gains over the pair it replaces is a near-repeat: high and xhigh are indistinguishable by every measurement available, they were run hours apart on the same stack, and they landed at 19 and 20. That is the closest thing to a run-to-run spread estimate this dial has, and a 13-point gap to max clears it comfortably. It is still not a replicate at max, which is the leg that would settle it. WHAT IS NOT CLAIMED: that xhigh is broken or ignored. It is accepted by the endpoint, echoed back applied, sits inside a dial measured real at the bottom, and on repo 1 it did think 54% harder than high. It simply did not convert that into fixes. Nor is it claimed that the ordering is exact - 19 against 20 is not evidence that xhigh beats high. THE HARNESS VERSION IS CAPTURED, NOT TYPED: this row records the muse --version read at run time, because muse self-updates and shipped a wrong version string to this board once already. THE SHIM IN THE PATH IS A METER AND NOT A ROUTE: Muse Code's JSON stream carries no usage payloads at all, so without it this row would have no billing record; pointed at Meta with the model rewrite and the provider pin both OFF it forwards the turn untouched and only counts. billing=LIST ESTIMATE, NOT A BILL: a subscription account exposes no per-run credits delta, so this is costed from the run's own token counts at the same Meta list rates the OpenRouter row was billed at, which keeps the Muse Code rows comparable. The benchmark repos were re-installed from the same lockfile inside Linux before the agent started, because the Windows install lays down win32 binaries Linux node cannot execute.", "note": "The third and last rung of Meta's first-party effort dial to be run on this board, and the one that turns a pair into a curve. *** THE WHOLE VALUE OF THIS DIAL SITS IN THE FINAL RUNG *** Three legs on one stack, one variable apart: high scores 19 of 105, xhigh scores 20, max scores 33. What is paid for them rises smoothly and the score does not: list cost runs 10.81, 15.58, 19.83 dollars and reasoning tokens run 178,300 then 225,766 then 286,242 - a step of almost exactly 27% at each rung. The first 27% more thinking buys ONE POINT, which is inside anything this board would call noise. The second 27% buys THIRTEEN. Going from high to xhigh costs 44% more and changes nothing measurable; going on to max costs another 27% and is the largest effort effect on this board. *** AND IT FALSIFIES THIS BOARD'S OWN MAGNITUDE RECEIPT A SECOND TIME, MORE PRECISELY THAN THE FIRST *** Receipt: effort-dial-probes/20260913-musespark13-meta-effort-magnitude.txt measured thinking volume at n=3 on one fixed prompt, found high, xhigh and max overlapping, and concluded they were a single saturated rung where paying for max over high buys nothing measurable. The bench says the top of this dial is TWO rungs, not one: high and xhigh land within a point of each other, and max is thirteen above both. So the probe got half of it right - high and xhigh ARE one rung, on the score as well as on the tokens - and put max in the wrong group, which is the half a buyer pays for. A token-volume sweep can group tiers and cannot price them. *** ON THE REAL WORKLOAD xhigh DOES NOT RELIABLY THINK MORE THAN high EITHER *** Per repo the sign flips: repo 1 at xhigh spent 178,161 reasoning tokens against high's 115,670, up 54%, while repo 2 at xhigh spent 47,605 against high's 62,630, down 24%. The combined +27% is one repo carrying it. So no measurement this board holds - toy-prompt tokens, bench tokens, or bench score - separates xhigh from high, and this row is published as the evidence for that rather than against it. *** THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT THE RUN *** Muse Code fires a reminder-observer side agent at a fixed low/high/medium the flag does not reach, so on a three-step probe roughly 4 of 10 calls carried the asked tier. The tier is asserted, validated in two layers and echoed back applied for the main loop, which is the agent that does the work. Receipt: effort-dial-probes/20260912-musecode-cli-effort-vs-wire.txt. Honesty profile: 8 of 45 on repo 1 and 12 of 60 on repo 2 with NO PARTIALS ON EITHER REPO - the first Muse row to manage that - ZERO claimed-only for the fifth Meta row running, and 7 genuine unplanted extras. Cost per strict fix is 78 cents here against 57 at high and 60 at max, which makes xhigh the worst-value rung of the three by the only metric that combines both columns.", "fixed_bugs": [ 16, 20, 25, 29, 30, 33, 34, 44, 47, 50, 60, 62, 71, 76, 83, 84, 92, 97, 98, 99 ], "turns": 356, "turns_repo1": 164, "turns_repo2": 192, "group_rank": 201 }, { "id": "Grok 4.7 (high) - mean of 3", "n_runs": 3, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 19.3, "repo1_fixed": 8.7, "repo2_fixed": 10.7, "partial": 2.3, "claimed_only": 0, "extras": 15.7, "false_positive_fixes": 0, "wall_min": 37.9, "cost_usd": 12.29, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration spanning 17 to 22 of 105. The Grok 4.6 row at this rung is a SINGLE run, so that comparison is three draws against one and is not a comparison of two distributions. Cost is a reconstructed FLOOR rather than a bill, and uses a per-session arithmetic the 4.6 rows on this board do not yet use; the note gives both figures. The runs spawned different numbers of subagent sessions on identical arm definitions, which the board's regime check cannot verify on this harness.", "note": "Grok 4.7 at high through xAI's own CLI, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 19, 22 and 17 of 105 (repo 1: 9, 9, 8 of 45 - repo 2: 10, 13, 9 of 60). Mean 19.3, spread 5. This supersedes the single-run row published earlier the same day at 19 of 105, which said the same thing on one draw. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 at high scored 23 of 105 (7 on repo 1, 16 on repo 2) - A SINGLE RUN, so the samples are not the same size and the comparison is three draws against one. Difference -3.7. That 4.6 number falls OUTSIDE the spread of these three runs, which is the strongest form this comparison can take while the other side is still a single draw - a difference the variance seen here does not explain, not a measured gap between two distributions. *** WHERE THIS RUNG SITS ON ITS OWN DIAL *** The same model at xhigh is 28.8 of 105 over four runs; this rung is 19.3 over three, a gap of +9.5. The dial as the board now shows it: low 11 (n=1), medium 30 (n=1), high 19.3 (n=3), xhigh 28.8 (n=4). For comparison Grok 4.6's dial ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, each of those from a single run. Whether that flatness is a property of the dial or a coincidence of two single draws is exactly what replicating these rungs was for. *** WHAT THE SPREAD ITSELF SAYS *** 5 points between the best and worst run of an IDENTICAL configuration - the three arms are clones and differ only in name. Any single-run number at this rung, including the one this row replaces, was carrying that much slack without being able to show it. Read any n=1 row on this board, from any vendor, with that in mind. *** COST AND TIME *** Reconstructed floors of $11.54, $12.02, $13.32 for a mean of $12.29; walls 40.5, 33.6, 39.6 minutes, mean 37.9. COST IS A FLOOR, not a bill: the grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates on the same 2/0.5/6 per MTok basis, and xAI bills a request whose prompt reaches 200K at DOUBLE those rates. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic that the 4.6 rows on this board still use it would read $12.29. Compare cost across rows only where both use the same arithmetic. *** GENUINE EXTRAS *** Real defects found in the repo that are not part of the planted set ran 18, 15 and 14 across the three runs, a mean of 15.7. Extras are not scored; they are reported because a model that finds real bugs nobody planted is doing something the planted-bug total cannot see. *** THE TIER WAS CHECKED, NOT ASSUMED, AND AT THE LOWER RUNGS THAT MATTERS MOST *** A zero-token ACP readback before each of these three launches returned reasoning_effort high for a requested high, with the served model named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except that readback - the run would be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run, one repo at a time, and only one grok run on the machine at a time - two grok agents share one login. Sessions spawned per leg (repo 1 / repo 2): run 1 1 and 1; run 2 1 and 1; run 3 1 and 1. That count varies run to run on identical arm definitions for this model, tracks cost closely, and has tracked score not at all. The board's automatic regime check parses Claude Code transcripts, so it returns NOT CHECKED for every grok row here and the session counts above are the manual substitute. Judge: blind Codex gpt-5.5 on all six legs, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 22, 25, 29, 30, 33, 34, 37, 39, 43, 45, 47, 50, 60, 62, 71, 76, 83, 84, 92, 95, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 3, "20": 3, "22": 1, "25": 2, "29": 1, "30": 3, "33": 1, "34": 1, "37": 3, "39": 2, "43": 2, "45": 3, "47": 2, "50": 2, "60": 3, "62": 2, "71": 3, "76": 3, "83": 2, "84": 3, "92": 1, "95": 1, "98": 2, "99": 3, "100": 3, "102": 2 }, "coverage_runs": 3, "turns": 397, "turns_repo1": 143, "turns_repo2": 254, "group_rank": 2 }, { "id": "DeepSeek V4.1 Flash (high effort)", "n_runs": 1, "model": "DeepSeek V4.1 Flash", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / DeepSeek API", "effort": "high", "effort_status": "verified", "date": "2026-09-10", "color": "#5A3EC8", "fixed": 19, "repo1_fixed": 9, "repo2_fixed": 10, "partial": 3, "claimed_only": 2, "extras": 4, "false_positive_fixes": 0, "wall_min": 26.1, "cost_usd": 0.31, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "The honesty profile is worse at this tier than at max, which is the opposite of the usual direction: 3 partials and 2 claimed-only here against 0 and 0 at max. At n=1 per tier that is a difference of a handful of bugs, not a demonstrated property of the tier.", "note": "The dial companion to the max row: same model, same first-party endpoint, same day, one tier down. It costs 5 strict fixes (19/105 against 24) and every one of them is on repo 1 - 9/45 against 14/45, while repo 2 lands on exactly the same 10/60. The tier is doing visibly less work for that: 26.1 min against 42.6, 160K output tokens against 240K, 57M cached reads against 82M, $0.31 against $1.08. high is DeepSeek's documented default, but it is requested explicitly here rather than left implicit, because on this endpoint the omitted field measured as neither high nor max.", "fixed_bugs": [ 13, 16, 20, 22, 29, 30, 39, 40, 43, 50, 60, 70, 71, 76, 83, 92, 97, 98, 99 ], "turns": 383, "turns_repo1": 153, "turns_repo2": 230, "group_rank": 2 }, { "id": "GPT-6 Luna (max effort) - mean of 2", "n_runs": 2, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 19, "repo1_fixed": 6.5, "repo2_fixed": 12.5, "partial": 1.5, "claimed_only": 0, "extras": 20.5, "false_positive_fixes": 0, "wall_min": 94.9, "cost_usd": 0.51, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 3 (18.3 of 105). This row reported 19.0 of 105 as the mean of 2 runs. The replicates are byte-identical re-runs; across 3 the runs span 17 to 21. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "Two runs of an identical configuration scoring 17 and 21 of 105 - spread 4. Read the mean with that width, not as a point, and note that two runs cannot bound variance well - the third is still to come. It also runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it compares cleanly to the GPT-6 Sol rows, which run the same build. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for every leg.", "note": "OpenAI's GPT-6 Luna at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. Published as the MEAN OF TWO INDEPENDENT RUNS. Runs scored 17 and 21 of 105 (repo 1: 6, 7 of 45 - repo 2: 11, 14 of 60). Mean 19.0, spread 4. *** THIS SUPERSEDES GPT-6 LUNA (MAX EFFORT) (17 OF 105) *** *** WHAT THIS ROW IS FOR: HOW MUCH SCORE SURVIVES A 20x PRICE CUT *** list price is $0.10 in and $0.50 out per million, twenty times cheaper than GPT-6 Sol ($2 / $10) on both sides of the meter. The board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max, GPT-6 Sol at 29.3 (mean of 3), and this model's own direct predecessor GPT-5.6 Luna at 31.3 (mean of 3). This row lands at 19.0. Read the gap against the price, not on its own. *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string gpt-6-luna, same effort, same timeout, same pinned executable, same code-mode setting, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** EXTRAS *** 22 and 19 genuine extras across the runs, mean 20.5 - real defects in the repos that are not part of the planted set. Extras are reported separately and never folded into the score. *** EXISTENCE WAS PROVED POSITIVELY, NOT INFERRED FROM A REJECTION *** the slug gpt-6-luna appears in the account's own model list and the pinned CLI accepted it and answered. The CLI returns 'The model is not supported when using Codex with a ChatGPT account' word for word for a deliberately fake slug, so a rejection is evidence about the client and never about whether a model exists. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows. It DOES compare cleanly to the GPT-6 Sol rows, which run the same pinned 0.155.1. *** WALL AND COST *** Walls 85.8, 104.0 minutes, mean 94.9; cost $0.52, $0.49, mean $0.51. COST IS A LIST ESTIMATE AND A FLOOR - token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 16, 20, 25, 29, 30, 33, 34, 39, 41, 43, 47, 49, 50, 62, 71, 76, 77, 78, 80, 82, 83, 84, 92, 98, 99, 102 ], "bug_hits": { "16": 1, "20": 1, "25": 2, "29": 1, "30": 2, "33": 1, "34": 1, "39": 2, "41": 1, "43": 1, "47": 2, "49": 2, "50": 2, "62": 2, "71": 1, "76": 2, "77": 2, "78": 1, "80": 1, "82": 1, "83": 2, "84": 2, "92": 2, "98": 1, "99": 1, "102": 1 }, "coverage_runs": 2, "turns": 270, "turns_repo1": 156, "turns_repo2": 115, "group_rank": 300 }, { "id": "GLM-5.3 Flash (max effort, Z.ai API)", "n_runs": 1, "model": "GLM-5.3 Flash", "vendor": "Z.ai", "local": false, "harness": "Claude Code / Z.ai API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-12", "color": "#55C9D8", "fixed": 19, "repo1_fixed": 10, "repo2_fixed": 9, "partial": 2, "claimed_only": 1, "extras": 4, "false_positive_fixes": 0, "wall_min": 58.4, "cost_usd": 0.94, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row at the same rung (17.7/105). This row is ONE run of three and it is the HIGHEST of the three: identical runs scored 19, 18 and 16. Nothing in it is retracted; it was the top of its own range. The replication could not be VERIFIED - Z.ai reports no token accounting - so the mean that replaces it is three runs believed to match rather than confirmed to.", "caveat": null, "note": "GLM-5.3 Flash on Z.ai's OWN Anthropic-compatible endpoint - no aggregator, no shim, the same first-party path as the full-size GLM-5.3 row, so the two sizes differ by model and by nothing else. It scores 19 of 105 for 95 cents in 58 minutes, which is the same 19 the full-size model scores on the same endpoint at the same asserted tier, for a sixteenth of the money. *** THIS NOTE TOLD READERS NOT TO CREDIT THE EFFORT DIAL. IT IS ANSWERED, 2026-09-13, AND THE DIAL GETS THE CREDIT *** A leg at low on this exact path scores 9 of 105 against this row's 19, with route, harness, model, prompt and repos all held fixed - so THE TIER IS WORTH TEN POINTS ON THIS MODEL and the OpenRouter row's 13 sits inside the range the dial alone covers. Superseded wording, kept verbatim rather than deleted: do not read the jump from the OpenRouter Flash row as an effort result; Flash scored 13 through OpenRouter and 19 here, and the tempting story is that the tier now binds; the control on this board argues against it, because the full-size model scored 19 on BOTH routes, so the same change of route and tier moved it by zero; each of these is one run and a 6-point move on n=1 does not clear the run-to-run spread measured elsewhere. THE CONTROL ITSELF IS NOT RETRACTED: the full-size GLM-5.3 has no tier pair on this board, only a route pair, so the zero it moved is a statement about its ROUTE and says nothing about its dial, which nobody has run. What was wrong was generalising one model's route result onto another model's tier. *** EFFORT max IS A SETTING, NOT A REQUEST *** Receipt: effort-dial-probes/20260912-glm53flash-zai-anthropic.txt. The endpoint validates output_config.effort - the field Claude Code actually puts on the wire - against a CLOSED enum: low, high and max return 200 while none, minimal, medium, xhigh and an invented bogus_zzz are all rejected HTTP 400 code 1210, and a bad model id is rejected separately as 1211, so the effort rejection is about the field rather than about a malformed request. An endpoint that refuses everything outside its list is not silently clamping what is inside it. Identical accept-list on both GLM sizes. NOTE THE FIELD, because it nearly produced the opposite verdict: a first pass probed top-level reasoning_effort, which is Z.ai's own OpenAI-compatible field name, got HTTP 200 on bogus_zzz and would have published this path as inert. output_config is what the endpoint reads and what the CLI sends. max is also Z.ai's documented default for this family, so a default run and a max run here are the same run - the row asserts the tier rather than relying on that. WHY THIS ROW IS LATE: the 2026-09-10 first-party pass tried Flash first and got HTTP 429 on this key, so it was written against the full-size model and Flash kept its aggregator row. The 429 is gone. *** AND THE LABEL MOVES WITH THE EVIDENCE: first_party -> verified_ceiling *** It shipped as first_party because the accept-list proved only that the endpoint READS the field. The low leg separates the levels on this serving path - 10 points of score and 3.1x the output tokens - which is what verified requires, and the same accept-list makes max the top rung that exists, which is what promotes it to a ceiling. Honesty profile: 10 of 45 on repo 1 with ONE claimed-only, 9 of 60 plus 2 partials on repo 2 with none, and 4 genuine extras across both. billing=LIST ESTIMATE, NOT A BILL: Z.ai publishes no balance or usage endpoint on this plan key, so no credits delta exists to read; costed at the same published Flash rates the OpenRouter Flash row was billed at, which keeps the two Flash rows comparable - those rates carry a launch discount that can lapse. ctx 1M, model string [1m].", "fixed_bugs": [ 13, 16, 20, 23, 25, 29, 30, 39, 43, 44, 50, 70, 71, 76, 83, 84, 92, 98, 99 ], "turns": 289, "turns_repo1": 100, "turns_repo2": 189, "group_rank": 100 }, { "id": "Sonnet 5.5 (low effort) - mean of 3", "n_runs": 3, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-09-30", "color": "#E68F74", "fixed": 19, "repo1_fixed": 6.7, "repo2_fixed": 12.3, "partial": 0.3, "claimed_only": 3.6, "extras": 8, "false_positive_fixes": 0, "wall_min": 8.5, "cost_usd": 5.73, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration scoring 20, 17 and 20 of 105 - spread 3. Read the mean with that width. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. Wall figures are contended.", "note": "Anthropic's Sonnet 5.5 at low effort through Claude Code on the first-party API, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 20, 17 and 20 of 105 (repo 1: 7, 6, 7 - repo 2: 13, 11, 13). Mean 19.0, range 3. *** THIS SUPERSEDES SONNET 5.5 (LOW EFFORT) (20 OF 105) *** *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string, effort, timeout, price table and blind judge; only the arm name differs, so the spread is run-to-run variance. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at low effort 22.3 (mean of 3) of 105. *** WALL AND COST *** walls 7.4, 9.7, 8.4 minutes, mean 8.5; cost $5.75, $6.20, $5.25, mean $5.73. *** GENUINE EXTRAS *** 5, 12 and 7 across the runs, mean 8.0, never folded into the score. *** HOW IT SPENT THE RUN *** On repo 1 it took 52, 39, 44 turns (the harness's own count); it first wrote its report 3, 5, 4 minutes in and its last model call came at minute 3, 5, 4. For scale: Opus 5.5 at low took 51-64 turns there over three runs of 4-5 minutes, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 67, 58, 39 turns (the harness's own count); it first wrote its report 4, 4, 4 minutes in and its last model call came at minute 4, 4, 4. For scale: Opus 5.5 at low took 58-86 turns there over three runs of 5-8 minutes, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 360,079 tokens; 0 compactions across 6 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. 3 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock.", "fixed_bugs": [ 13, 20, 22, 25, 30, 39, 43, 45, 46, 47, 49, 50, 60, 71, 76, 77, 82, 83, 84, 92, 98, 99, 100 ], "bug_hits": { "13": 2, "20": 3, "22": 1, "25": 2, "30": 3, "39": 3, "43": 3, "45": 3, "46": 2, "47": 1, "49": 2, "50": 3, "60": 2, "71": 3, "76": 3, "77": 3, "82": 2, "83": 1, "84": 3, "92": 3, "98": 3, "99": 3, "100": 3 }, "coverage_runs": 3, "turns": 100, "turns_repo1": 45, "turns_repo2": 55, "group_rank": 504 }, { "id": "Muse Spark 1.3 (high effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "high", "effort_status": "verified", "date": "2026-09-13", "color": "#E57EA2", "fixed": 19, "repo1_fixed": 8, "repo2_fixed": 11, "partial": 2, "claimed_only": 0, "extras": 10, "false_positive_fixes": 0, "wall_min": 64.5, "cost_usd": 10.81, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same rung, which landed on the SAME score (18.7, rounding to 19). Nothing here is corrected by arithmetic; this row is one of three runs that scored 19, 17 and 20, and it is kept because a single run is no longer how this rung is reported. The one claim that does not survive is in the caveat below: the high/xhigh pair agreeing within a point was offered as a run-to-run spread estimate, and the real spread at this tier is 3 points, so that pair never established it.", "caveat": "One run at this tier. The 14-point gap to max was n=1 against n=1 when this row shipped; the xhigh leg has since landed at 20, so TWO independent runs at the top-but-one sit within a point of each other while the single max run sits thirteen above both. high and xhigh are indistinguishable by every measurement this board holds, which makes that pair the closest thing to a run-to-run spread estimate this dial has. It is still not a replicate at max, which is the leg that would turn a strong observation into a finding. Note also what is NOT claimed: that high is a bad setting. It is this endpoint's default, it costs 45% less than max and finishes in under two thirds the wall clock, and on a board where most rows sit below 30 it is an ordinary result. What the pair shows is that on THIS stack the dial is worth paying for, which is the opposite of the Qwen and GLM findings and is why both belong on the same page. THE HARNESS VERSION IS CAPTURED, NOT TYPED: this row records the muse --version read at run time, because muse self-updates and shipped a wrong version string to this board once already. THE SHIM IN THE PATH IS A METER AND NOT A ROUTE: Muse Code's JSON stream carries no usage payloads at all, so without it this row would have no billing record; pointed at Meta with the model rewrite and the provider pin both OFF it forwards the turn untouched and only counts. billing=LIST ESTIMATE, NOT A BILL: a subscription account exposes no per-run credits delta, so this is costed from the run's own token counts at the same Meta list rates the OpenRouter row was billed at, which keeps the Muse Code rows comparable. The benchmark repos were re-installed from the same lockfile inside Linux before the agent started, because the Windows install lays down win32 binaries Linux node cannot execute.", "note": "Meta's own agent on Meta's own endpoint at high - which is also this endpoint's MEASURED default, since omitting the effort field returns an echo of high. So this row is what the stack does out of the box first-party, and it exists to answer a question the max row could not. *** IT SETTLES THE ATTRIBUTION, AND AGAINST THE READING THIS BOARD PUBLISHED FIRST *** The max row scored 33/105 against an OpenRouter sibling's 18 with the ROUTE and the EFFORT having moved together, and its note warned against crediting the dial. The sibling is now known to have run at high as well - an unflagged Muse Code sends reasoning.effort=high and that route applies it - so these three rows differ one variable at a time: OpenRouter at high scores 18, Meta first-party at high scores 19, Meta first-party at max scores 33. THE ROUTE IS WORTH ABOUT A POINT. THE TIER IS WORTH FOURTEEN. That is the largest effect an effort setting has produced on this board, and it is the reverse of what the GLM-5.3 first-party pair showed, where the same kind of move was worth zero. *** AND IT FALSIFIES A PREDICTION THIS BOARD PUBLISHED IN WRITING *** Receipt: effort-dial-probes/20260913-musespark13-meta-effort-magnitude.txt said that because high and max could not be separated in thinking tokens at n=3 on a fixed prompt (means 9385 against 12655, ranges overlapping), a bench leg at high should land near 33. It landed at 19. On the real workload the tiers are not close: this row spent 178,300 reasoning tokens against the max row's 286,242, a 61% gap the toy prompt could not see. READ THAT AS A LIMIT ON MAGNITUDE PROBES GENERALLY - a token-volume sweep shows whether a dial is CONNECTED, not what it is WORTH, and this board will not treat one as evidence for the other again. *** THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT THE RUN *** Muse Code fires a reminder-observer side agent at a fixed low/high/medium the flag does not reach, so on a three-step probe roughly 4 of 10 calls carried the asked tier. The tier is asserted, validated in two layers and echoed back applied for the main loop, which is the agent that does the work. Receipt: effort-dial-probes/20260912-musecode-cli-effort-vs-wire.txt. Honesty profile: 8 of 45 on repo 1 with no partials, 11 of 60 plus 2 partials on repo 2, ZERO claimed-only on either repo - the fourth Meta row in a row with a clean claim record - and 10 genuine unplanted extras.", "fixed_bugs": [ 16, 20, 22, 25, 30, 33, 34, 45, 47, 49, 60, 62, 71, 76, 83, 92, 98, 99, 102 ], "turns": 221, "turns_repo1": 141, "turns_repo2": 80, "group_rank": 202 }, { "id": "Grok 4.7 (high)", "n_runs": 1, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 19, "repo1_fixed": 9, "repo2_fixed": 10, "partial": 3, "claimed_only": 0, "extras": 18, "false_positive_fixes": 0, "wall_min": 40.5, "cost_usd": 11.53, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-21 by the n=3 row at the same configuration (19.3 of 105). The three runs scored 19, 22 and 17, so this single draw at 19 sat inside a 5-point spread it could not show. Nothing it claimed is retracted; it was published as an n=1 row and read as one. Kept rather than deleted because it is the receipt for what the board showed for part of 2026-09-21.", "caveat": "ONE RUN, not a mean. The same model at its top rung spanned 25 to 31 of 105 across three runs, so read this number with several points of slack in either direction. Cost is a reconstructed FLOOR rather than a bill, and it uses a per-session arithmetic the older grok rows on this board do not yet use; the note gives both figures.", "note": "Grok 4.7 at high through xAI's own CLI - the CLI DEFAULT rung - what someone gets who never touches the effort setting, and the only number on this dial that describes the out-of-the-box experience. 19 of 105: 9 of 45 on repo 1 and 10 of 60 on repo 2, 18 genuine extras, 40.5 minutes, a reconstructed floor of $11.53. ONE RUN. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 (high) scored 23.0 of 105 (7 on repo 1, 16 on repo 2), also a single run, for a difference of -4. Both sides are n=1, so this is one draw against one draw. The scale of the doubt is not a guess: three runs of the 4.6 configuration one rung up spanned 27 to 30 of 105, and three runs of this model at that rung spanned 25 to 31 - so a gap of a few points between two single runs is inside what the same configuration produces run to run. *** WHERE IT SITS ON ITS OWN DIAL *** This model at xhigh is 28.8 of 105, published as a mean of four. This rung is 19, a single run. The 4.6 dial for comparison ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, which is the most interesting thing on it, and the low rung is where that dial had real room. Whether either shape repeats here is a question the rungs answer together: read this row beside its siblings rather than on its own. *** THE TIER WAS CHECKED, NOT ASSUMED, AND FOR THE LOWER RUNGS THAT MATTERS MORE *** A zero-token ACP readback before launch returned reasoning_effort high for a requested high. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except this readback - the run would simply be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. The served model is named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. *** COST IS A FLOOR *** The grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates, the same 2/0.5/6 per MTok basis. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic the other grok rows still use it would read $11.54. It is a floor twice over: xAI bills a request whose prompt reaches 200K at DOUBLE those rates. *** RECEIPTS *** Both legs exited 0. Legs SEQUENTIAL, one repo at a time, uncontended - nothing else ran alongside. The agent spawned 1 session(s) on repo 1 and 1 on repo 2 (57 and 73 inference requests); that count varies run to run on identical arm definitions for this model, and it tracks cost far more closely than score. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 22, 25, 30, 37, 39, 45, 47, 60, 62, 71, 76, 84, 98, 99, 100, 102 ], "turns": 384, "turns_repo1": 150, "turns_repo2": 234, "group_rank": 2 }, { "id": "GLM-5.3 (max effort, Z.ai API)", "n_runs": 1, "model": "GLM-5.3", "vendor": "Z.ai", "local": false, "harness": "Claude Code / Z.ai API", "effort": "max", "effort_status": "first_party", "date": "2026-09-10", "color": "#14B5C9", "fixed": 19, "repo1_fixed": 9, "repo2_fixed": 10, "partial": 0, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 40.4, "cost_usd": 15.93, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "One run on each side of the comparison. Two single runs landing on the same total is not a demonstration that the paths are equivalent - the board has watched one model at one setting score 16, 13 and 17 - and the matching totals here are best read as 'no effect large enough for one run to see' rather than 'no effect'. The wall and cost gaps are the more robust half of this row, because they do not depend on the judge. This arm also ran twice: the first attempt died at 120 turns when the Z.ai balance ran out, and that leg was voided rather than scored. The figures here are the complete re-run.", "note": "GLM-5.3 on Z.ai's own API at max effort scores 19/105 - the identical total to the board's existing GLM-5.3 row, which ran through OpenRouter at an inert default effort. The splits differ slightly (9/45 + 10/60 here against 8/45 + 11/60 there) and the total does not. Two variables moved between those two rows at once, the serving path AND the requested tier, and neither shifted the score. What did shift is everything else: 40.4 minutes against 66.7, and $15.93 against $19.73. So the first-party route is 1.65x faster and cheaper for the same result. On the tier itself: Z.ai's endpoint 400-rejects an invented effort value and names its accept-list (none, minimal, low, medium, high, xhigh, max), so unlike an aggregator that shrugs and returns 200, this route demonstrably READ the field - but a field being read is not the same as a dial being turned, and no magnitude probe was run here, so this row claims an accepted tier and not a measured one.", "fixed_bugs": [ 13, 16, 20, 27, 29, 30, 39, 40, 43, 49, 50, 71, 76, 83, 97, 98, 99, 100, 102 ], "turns": 344, "turns_repo1": 116, "turns_repo2": 228, "group_rank": 0 }, { "id": "GLM-5.3", "n_runs": 1, "model": "GLM-5.3", "vendor": "Z.ai", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-25", "color": "#14B5C9", "fixed": 19, "repo1_fixed": 8, "repo2_fixed": 11, "partial": 1, "claimed_only": 1, "extras": 4, "false_positive_fixes": 0, "wall_min": 66.7, "cost_usd": 19.73, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Aggregator run at an inert default effort. The Sep 10 row runs the same model on Z.ai's own API with the tier asserted against an endpoint that rejects invented tiers, scores the same 19/105, and does it in 40.4 min for $15.93 against 66.7 min and $19.73 - so the first-party row replaces this one.", "caveat": null, "note": "Run as the behavioural control for the Ox Alpha stealth slug, whose wire fingerprint matches this model's serving stack. Their claim lists overlapped heavily, but only 10 of their 25 graded fixes are the same bugs.", "fixed_bugs": [ 16, 20, 25, 26, 29, 30, 40, 43, 49, 60, 61, 71, 76, 83, 92, 97, 98, 99, 102 ], "turns": 338, "turns_repo1": 158, "turns_repo2": 180, "group_rank": 6 }, { "id": "Qwen3.8-Max", "n_runs": 1, "model": "Qwen3.8-Max", "vendor": "Alibaba", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-03", "color": "#E8A020", "fixed": 19, "repo1_fixed": 5, "repo2_fixed": 14, "partial": 0, "claimed_only": 1, "extras": 6, "false_positive_fixes": 0, "wall_min": 148.2, "cost_usd": 33.3, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded by the Sep 11 first-party run: same model on Alibaba's own endpoint instead of Claude Code -> local proxy -> OpenRouter, scoring 28/105 for $26.29 against this row's 19/105 for $33.30. This row billed 12.94% of its prompt tokens as uncached input against the first-party run's 0.0019%, so it priced the route rather than the model.", "caveat": null, "note": "CORRECTED 2026-09-12 (4): EFFORT IS INERT ON THIS ROW, measured at last. It shipped verified_ceiling, was corrected to first_party on 09-11, and is now inert_default with effort 'default' - the same label every other aggregator-routed row on this board carries. A nine-runs-per-condition probe on the exact surface this arm used (OpenRouter's Anthropic-compatible /v1/messages, receipts results/effort-dial-probes/20260912-qwen38max-openrouter-anthropic.txt and 20260912-qwen38max-all-n9.txt) puts reasoning.effort=low at mean 13,017 output tokens (7,897-16,402), reasoning.effort=high at 14,815 (11,524-20,237), and NO effort field at all at 13,622 (9,179-19,757) - a 1.14x spread between the tiers against a 2.15x worst within-condition spread, with every range overlapping every other. The field does nothing; the run sat at Alibaba's own default either way. SELF-CORRECTION, same day: for a few hours this note also said the shim had caused the problem by deleting Anthropic's native 'thinking' field to install 'reasoning.effort', on the strength of that field measuring very differently (mean 5,468). That rested on ONE budget value, which can only show a field is read, not that it binds. A three-value probe (receipt results/effort-dial-probes/20260912-qwen38max-budget-scaling-openrouter.txt, n=3 each) put budget_tokens=2,000 at mean 5,627 output tokens, 8,000 at 4,735 and 24,000 at 4,111 - twelve times the budget moved the mean 0.73x, the wrong way, inside a 1.95x worst within-condition spread. So budget_tokens is a switch rather than a dial on this path, and the switch SUPPRESSES: sending it in any form costs about two thirds of the model's thinking against the 13,622 no-field baseline. That makes the shim's pop the configuration producing the MOST thinking available here, not a discarded control - with 'thinking' gone and reasoning.effort inert, the request fell through to the endpoint default, which is exactly what inert_default records. The label is unaffected; only the causal story changed, and the shims were reverted to dropping the field. A control on Alibaba's own endpoint shows native thinking landing far higher there and fully overlapping Alibaba's own no-field baseline, so the suppression is a property of the aggregator hop and not of the model. RE-MEASURED AT n=6 AND THEN n=9 the same day, because a claim this size should not rest on three samples (receipt results/effort-dial-probes/20260912-qwen38max-all-n9.txt, 24 fresh concurrent calls pooled with the originals and the n=6 top-ups; every condition on both routes now stands at nine runs, so no verdict rests on a smaller sample than the one it is compared against): through OpenRouter budget_tokens=24,000 gives mean 4,967 (3,568-7,637) against a no-field baseline of 13,622 (9,179-19,757); straight to Alibaba the same field gives 13,168 (9,185-15,977) against 14,379 (11,586-17,442). All verdicts survive - cross-route 2.65x against a 2.15x worst within-cell spread, ranges disjoint, the field costing 64% of the thinking through the hop and nothing at all direct. The effort WORD reads more inert at nine runs than at three: low-vs-high means 1.14x through OpenRouter and 1.04x direct, ranges fully overlapping. The bigger samples did move one number: the OpenRouter no-field baseline measured 15,714 at n=3, 13,339 at n=6 and 13,622 at n=9 - three samples had overestimated it by 15%, and the six- and nine-run readings agree to 2%. Quote the nine-run figures only. Ranges can only widen with n, so each pass was a test the finding could fail. Still at n=3 and flagged as such: the three-value budget sweep, whose top-up was attempted the same day and blocked when the OpenRouter account ran out of credits mid-series - read the RANKING of the three budgets as unmeasured; the suppression claim does not depend on it. The scores, cost and wall are unchanged. EARLIER CORRECTIONS, 2026-09-11, on three counts, while adding the Qwen3.8-Flash rows. (1) HARNESS. This row was published as 'Claude Code / Alibaba API'. It never called Alibaba's API. The arm ran through a local pass-through shim with upstream=OpenRouter's qwen3.8-max slug, so the path was Claude Code -> local proxy -> OpenRouter -> Alibaba, exactly like the Kimi K3, Ox Alpha and GLM-5.3 rows, all of which correctly say 'Claude Code / OpenRouter'. This row was the only OR-routed row on the board claiming a direct vendor API. (2) EFFORT STATUS. It was published verified_ceiling. No probe receipt exists - the only Qwen file in results/effort-dial-probes is the 2026-09-11 Flash one. The claim rested on QwenCloud's documented enum (low|medium|xhigh, default xhigh, high->xhigh) plus a check that thinking was live (248 thinking blocks across the raw logs). Under this board's own vocabulary that is first_party - a documented tier taken at face value - because verified_ceiling requires a probe that SEPARATES the levels. The row's own note also argued that Alibaba validates effort 'while OR accepts any string and applies none', which cannot both be true of a request routed through OR; whether the effort field survived the proxy and the aggregator to reach Alibaba was never established. (3) COST, $31.10 -> $33.30. The arm modelled cache writes as FREE. OpenRouter lists qwen3.8-max-0902 cache writes at $2.50/MTok and this run wrote 879,902 cache-creation tokens. A sweep of OR's listing found Qwen is the ONLY family on this board it charges a cache-write fee for - DeepSeek, Z.ai, Moonshot, Tencent and Meta publish none - so $0.00 is right on every other aggregator-routed arm and was wrong only here. The SCORES are unchanged; only the two labels and the cost move. And a caveat larger than the correction: 12.94% of this row's prompt tokens (8,282,777 of them, $16.57 - HALF the row) were billed as uncached input, against 0.002% on the two Flash rows running the same harness against Alibaba's own endpoint. That ~5,000x gap in cache-miss rate is a property of the proxy-plus-aggregator path, not of the model, so read this row's cost as a cost OF THAT PATH rather than a price for Qwen3.8-Max. NOT re-probed on purpose: the Aug 3 arm used an unlisted qwen3.8-max slug and what OpenRouter lists now is qwen3.8-max-0902, a dated snapshot, so a probe today would measure a different serving path than this row shipped on. The 2026-09-11 Flash finding does not transfer either (different model AND different path).", "fixed_bugs": [ 20, 25, 27, 30, 43, 50, 51, 60, 61, 66, 69, 70, 71, 76, 78, 83, 97, 98, 99 ], "turns": 303, "turns_repo1": 154, "turns_repo2": 149, "group_rank": 6 }, { "id": "Muse Spark 1.3 (high effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "high", "effort_status": "verified", "date": "2026-09-14", "color": "#E57EA2", "fixed": 18.7, "repo1_fixed": 8.7, "repo2_fixed": 10, "partial": 1.7, "claimed_only": 1, "extras": 19.3, "false_positive_fixes": 0, "wall_min": 75.6, "cost_usd": 12.94, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 17 to 20. This row SUPERSEDES the single-run high row (19/105) published 2026-09-13, which landed on the SAME number - the supersession is about method, not a correction, and the only claim it withdraws is that row's use of the high/xhigh pair as a spread estimate. Every figure on this row is a MEAN of the three runs, carried to one decimal - 18.7 of 105. Both rungs above this one have since been replicated as well: xhigh 20.3 (range 9) and max 33.7 (range 2). The 15-point step from high's mean to max's is measured at n=3 on both ends.", "note": "THIS ENDPOINT'S MEASURED DEFAULT RUNG, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 19, 17 and 20 of 105 (repo 1: 8, 10, 8 - repo 2: 11, 7, 12). *** THIS ONE CONFIRMS THE ROW IT WAS RUN TO CHECK *** The three legs average 18.7, which rounds to the 19 the single high run published on 2026-09-13. That is the opposite of what the same experiment did one and two rungs down, where the n=3 means reversed the published ordering of low and medium. It is worth saying plainly: replication is not a debunking tool, and this time the published number survived intact. *** WHAT THE THREE REPLICATED RUNGS SHOW, AND WHAT THEY DO NOT *** The observed ranges are low 12/13/4 (9), medium 9/15/15 (6), high 19/17/20 (3). That looks like a spread narrowing as the dial rises, and THIS ROW DOES NOT CLAIM IT. A range taken from three draws is a very noisy estimate of spread - three runs can fall 9, 6, 3 by chance alone - and three such estimates come nowhere near establishing a trend in variance, which would need far more runs per rung than this board is going to buy. Recorded as an OBSERVATION TO WATCH, not a finding. In particular, do NOT read it as 'a single run near a model's ceiling is safer than one at the bottom of its dial' - that is the same n=1 reasoning this experiment exists to reject, moved up a level, and no row on this board should be discounted or trusted on the strength of it. *** AND EFFORT BUYS RELIABILITY, NOT A LONGER TAIL OF LUCK *** Counting per BUG across each rung's three runs: low fixed 21 distinct bugs at least once and 2 of them every time, medium 23 ever and 4 always, high 30 ever and 8 always. The share fixed in ALL THREE runs climbs 10 to 17 to 27 per cent, while the fixed-exactly-once count barely moves (15, 11, 12). The direction that suggests is bugs PROMOTED from once to twice-or-always rather than a longer tail of one-off luck. SIZE OF THE EVIDENCE: the always column is 2, 4 and 8 bugs, counted over three runs apiece, so the percentages are illustrative and the direction is suggestive - neither is a measured reliability. The Coverage view shades each tick by the observed hit rate, so the raw counts are readable on the row itself rather than only the summary. *** THE COST OF A FIXED FLAG IS NOT FIXED *** Same arm, same tier, three runs: 10.81, 12.70 and 15.29 dollars, and 64.5, 85.8 and 76.7 minutes - a 41 per cent cost spread with nothing changed. Calls ran 373/226, 439/261 and 293/418 across the two repos. The monotone turns-track-score pattern visible in low's repo-1 legs does NOT reproduce here: repo 2's busiest leg scored BEST and repo 1's busiest scored two above its quietest. That supports the weaker claim already published at the bottom of this dial - the effort flag fixes a request field, not an amount of work, and the work performed varies by about a third inside one tier - and it does not support turns predicting score. *** WHAT THIS RETRACTS *** The single-run high row's caveat called high (19) and xhigh (20) sitting within a point of each other 'the closest thing to a run-to-run spread estimate this dial has'. With a real replicate the high spread is 3 points, so two rows one point apart were always inside one tier's noise and never carried that evidence. The CONCLUSION survives - high and xhigh remain indistinguishable - but it now rests on a measured spread instead of on two single runs agreeing by luck. All six legs were clean: the Linux install returned 0, the shim logged zero errors, every leg exited 0, and no leg carries a handicap note. Honesty profile across the three runs: 3 claimed-only in six legs (0, 2 and 1 by run) and 10, 20 and 28 genuine unplanted extras. No planted bug fell for the first time in either new leg, so the survivor count is unchanged. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR: Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control. billing=LIST ESTIMATE, NOT A BILL: a subscription account with no per-run credits delta, costed at the same Meta list rates as the other first-party rows, which is what keeps them comparable. ctx 1M.", "fixed_bugs": [ 16, 20, 21, 22, 25, 28, 30, 33, 34, 39, 43, 44, 45, 47, 49, 50, 60, 62, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "16": 3, "20": 3, "21": 1, "22": 3, "25": 3, "28": 1, "30": 3, "33": 2, "34": 2, "39": 1, "43": 2, "44": 1, "45": 1, "47": 2, "49": 1, "50": 1, "60": 2, "62": 1, "71": 2, "76": 2, "77": 1, "82": 1, "83": 2, "84": 2, "92": 3, "97": 1, "98": 2, "99": 3, "100": 1, "102": 3 }, "coverage_runs": 3, "turns": 268, "turns_repo1": 142, "turns_repo2": 126, "group_rank": 202 }, { "id": "GPT-6 Luna (max effort) - mean of 3", "n_runs": 3, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 18.3, "repo1_fixed": 6.3, "repo2_fixed": 12, "partial": 2, "claimed_only": 0, "extras": 22.7, "false_positive_fixes": 0, "wall_min": 99.2, "cost_usd": 0.52, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "Three runs of an identical configuration scoring 17, 21 and 17 of 105 - spread 4. Read the mean with that width, not as a point. It also runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it compares cleanly to the GPT-6 Sol rows, which run the same build. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for every leg.", "note": "OpenAI's GPT-6 Luna at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. Published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 17, 21 and 17 of 105 (repo 1: 6, 7, 6 of 45 - repo 2: 11, 14, 11 of 60). Mean 18.3, spread 4. *** THIS SUPERSEDES GPT-6 LUNA (MAX EFFORT) - MEAN OF 2 (19.0 OF 105) *** *** WHAT THIS ROW IS FOR: HOW MUCH SCORE SURVIVES A 20x PRICE CUT *** list price is $0.10 in and $0.50 out per million, twenty times cheaper than GPT-6 Sol ($2 / $10) on both sides of the meter. The board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max, GPT-6 Sol at 29.3 (mean of 3), and this model's own direct predecessor GPT-5.6 Luna at 31.3 (mean of 3). This row lands at 18.3. Read the gap against the price, not on its own. *** THE REPLICATES ARE BYTE-IDENTICAL TO THE FIRST RUN *** same model string gpt-6-luna, same effort, same timeout, same pinned executable, same code-mode setting, same price table, same blind judge. Only the arm name differs, so the spread above is run-to-run variance and nothing else. *** EXTRAS *** 22, 19 and 27 genuine extras across the runs, mean 22.7 - real defects in the repos that are not part of the planted set. Extras are reported separately and never folded into the score. *** EXISTENCE WAS PROVED POSITIVELY, NOT INFERRED FROM A REJECTION *** the slug gpt-6-luna appears in the account's own model list and the pinned CLI accepted it and answered. The CLI returns 'The model is not supported when using Codex with a ChatGPT account' word for word for a deliberately fake slug, so a rejection is evidence about the client and never about whether a model exists. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows. It DOES compare cleanly to the GPT-6 Sol rows, which run the same pinned 0.155.1. *** WALL AND COST *** Walls 85.8, 104.0, 107.7 minutes, mean 99.2; cost $0.52, $0.49, $0.57, mean $0.53. COST IS A LIST ESTIMATE AND A FLOOR - token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 16, 20, 25, 29, 30, 33, 34, 39, 40, 41, 43, 44, 47, 49, 50, 62, 71, 76, 77, 78, 80, 82, 83, 84, 92, 98, 99, 100, 102 ], "bug_hits": { "16": 2, "20": 2, "25": 3, "29": 1, "30": 3, "33": 1, "34": 1, "39": 2, "40": 1, "41": 1, "43": 1, "44": 1, "47": 3, "49": 3, "50": 2, "62": 3, "71": 2, "76": 3, "77": 3, "78": 1, "80": 1, "82": 1, "83": 3, "84": 3, "92": 2, "98": 2, "99": 2, "100": 1, "102": 1 }, "coverage_runs": 3, "turns": 286, "turns_repo1": 153, "turns_repo2": 133, "group_rank": 300 }, { "id": "Haiku 5.5 (xhigh effort) - mean of 3", "n_runs": 3, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "xhigh", "effort_status": "first_party", "date": "2026-10-07", "color": "#EBA691", "fixed": 18.3, "repo1_fixed": 9, "repo2_fixed": 9.3, "partial": 0.3, "claimed_only": 0.7, "extras": 6.7, "false_positive_fixes": 0, "wall_min": 24.1, "cost_usd": 4.8, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": true, "superseded": null, "caveat": "Mean of three runs of one configuration (16, 19 and 20 of 105). Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers - most requests land in the upper tier under Claude Code. Effort xhigh was read off the wire because the CLI does not yet recognise the model; Claude Code's title call runs at high.", "note": "Anthropic's Claude Haiku 5.5 at xhigh effort through Claude Code on the first-party API, released 2026-10-07 and run the same day. MEAN OF THREE RUNS of one configuration: 16, 19 and 20 of 105 (repo 1: 6, 8 and 13 of 45 - repo 2: 10, 11 and 7 of 60), mean 18.3. Genuine unplanted extras 10, 5 and 5; claimed-only fixes 2, 0 and 0. Walls 22.7, 20.5 and 29.1 minutes; cost $4.50, $4.30 and $5.59. *** EFFORT, READ OFF THE WIRE *** Claude Code 2.1.292 does not know this model yet (it logs it as unrecognized), so the request was captured rather than the CLI taken at its word: the main loop sends effort xhigh with adaptive thinking to claude-haiku-5-5, with the 1M-context header, and the API accepts it. Claude Code's one per-session title call runs at high, which the effort flag does not reach; the tier describes the main loop. A first-party tier, accepted, not magnitude-probed; not the ceiling (max exists). *** COST IS PRICED PER REQUEST *** Haiku 5.5 is billed by each request's prompt length: $0.10 input / $0.20 1-hour cache write / $0.01 cache read / $0.50 output per million tokens up to a 100,000-token prompt, and $0.50 / $1 / $0.05 / $2.50 above it, every token of the request at its prompt's tier. Claude Code's instructions and tools alone are about 50,000 tokens, so the conversation crosses the line within a few turns: 552 of the 587 requests across the six legs were priced at the upper tier. Each request was priced from Claude Code's own session record, because the CLI's cost table has no entry for this model. 98.6% of prompt tokens were cache reads. A list estimate, not a bill. *** RECEIPTS *** All six legs exited 0, run strictly one at a time on one account. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 16, 20, 22, 23, 25, 26, 27, 28, 29, 30, 33, 34, 39, 40, 43, 45, 49, 50, 60, 61, 71, 76, 77, 83, 84, 92, 97, 99, 100, 102 ], "bug_hits": { "16": 1, "20": 3, "22": 1, "23": 1, "25": 3, "26": 1, "27": 1, "28": 1, "29": 1, "30": 3, "33": 1, "34": 1, "39": 1, "40": 2, "43": 3, "45": 3, "49": 3, "50": 1, "60": 1, "61": 1, "71": 2, "76": 3, "77": 3, "83": 2, "84": 3, "92": 1, "97": 1, "99": 3, "100": 3, "102": 1 }, "coverage_runs": 3, "turns": 246, "turns_repo1": 104, "turns_repo2": 142, "group_rank": 701 }, { "id": "MiMo-V2.6-Pro (first-party)", "n_runs": 1, "model": "MiMo-V2.6-Pro", "vendor": "Xiaomi", "local": false, "harness": "Claude Code / Xiaomi first-party", "effort": "enabled", "effort_status": "verified", "date": "2026-09-22", "color": "#1A1AA6", "fixed": 18, "repo1_fixed": 9, "repo2_fixed": 9, "partial": 0, "claimed_only": 0, "extras": 4, "false_positive_fixes": 0, "wall_min": 74.9, "cost_usd": 0.59, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. The OpenRouter default row on this same model scored 2 points higher (20 vs 18 of 105, both single runs) - inside the noise this board has already measured elsewhere (widest observed spread from an identical config: 6 of 105), and expected here since both rows almost certainly ran with reasoning already on. Effort is `enabled`, this model's own vocabulary for its one and only reasoning toggle - there is no higher tier to assert. Cost is a list-price estimate, not a checked bill. The wall figure is contended.", "note": "Xiaomi's MiMo-V2.6-Pro driven by Claude Code through an Anthropic-API shim onto Xiaomi's OWN first-party endpoint - no aggregator hop at all, unlike the default row published earlier the same day. 18 of 105: 9 of 45 on repo 1 and 9 of 60 on repo 2, 4 genuine extras, 74.9 minutes, $0.59. ONE RUN. *** THIS MODEL HAS NO GRADUATED EFFORT DIAL, ON ANY ROUTE - CONFIRMED TWO WAYS *** Xiaomi's own docs document exactly one reasoning control for this model: `thinking.type` = enabled or disabled, a binary toggle, not a low/medium/high/xhigh scale. Confirmed directly, not just from docs: a 2-call test against this same endpoint found thinking=disabled gives 0 reasoning tokens (542 completion tokens total) and thinking=enabled gives 2,375 chars of reasoning content (1,696 completion tokens). This also explains the earlier OpenRouter finding on the default row - reasoning.effort was accepted but inert there (n=3 per condition, low/high/no-field ranges fully overlapping) because OpenRouter was normalising onto this SAME binary toggle, and no effort value it sent could ever separate on a route with only two states. *** THIS ROW ASSERTS THE TOGGLE EXPLICITLY *** Every call carries thinking={type:enabled}, forced by the shim regardless of what the harness sent, verbatim Claude-shaped (the field name and structure this endpoint documents). Labelled `enabled`, not `max` or `high`: borrowing a graduated-tier name would imply a scale this model does not have. *** COST IS A LIST-PRICE ESTIMATE, NOT A BILL *** $0.59 is token counts against Xiaomi's own published per-token rate (0.435 in / 0.0036 cache-read / 0.87 out per MTok) - real usage at a real rate, but this platform's balance/usage API was not checked the way OpenRouter's credits-delta was for the other row, so it is not called a bill. *** RECEIPTS *** Key is a budget-capped ($5) first-party test key. Both legs exited 0, run SEQUENTIALLY, one repo at a time. The run shared the laptop with benchmark legs from other vendors, which the wall marker on this row discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it. Second measurement of this model, published as its own row rather than averaged into the first: route and an explicitly-asserted effort are configuration, not noise.", "fixed_bugs": [ 16, 20, 25, 26, 27, 28, 29, 30, 43, 47, 49, 50, 60, 62, 71, 76, 99, 100 ], "turns": 288, "turns_repo1": 113, "turns_repo2": 175, "group_rank": 8 }, { "id": "Hy4 Preview", "n_runs": 1, "model": "Hy4 Preview", "vendor": "Tencent", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-28", "color": "#7C2D12", "fixed": 18, "repo1_fixed": 9, "repo2_fixed": 9, "partial": 0, "claimed_only": 3, "extras": 2, "false_positive_fixes": 0, "wall_min": 67.5, "cost_usd": 3.13, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "Tencent Hunyuan 4 preview (MoE, 49B active of 770B), benchmarked day-one via OpenRouter. 18/105 for $3.13 real OpenRouter bill. On repo 1 it claimed 10 fixes and 9 verified against the answer key. Single host (Tencent Cloud); day-one 429/404s absorbed by proxy retries. EFFORT CORRECTED 2026-09-12, from `high / first_party` to `default / inert_default`. The day-one row took Tencent's documentation at its word - reasoning_effort is a BINARY switch, `high` (the default) or `no_think`, so there is no higher tier to ask for - and published the tier the arm REQUESTED. Probing the route it actually ran on says the request never arrived: at n=3 per condition on OpenRouter's /v1/messages, both spellings overlap the no-field baseline, the baseline is the HIGHEST band of the five (mean 13,520 output tokens against 8,243 for nested high), and `bogus_zzz` returns HTTP 200 on both spellings while an invented model id 400s - no validation, so nothing downstream reads the field. Receipt: results/effort-dial-probes/20260912-hy4preview-openrouter.txt. The row is still at the model's ceiling, but for a different reason than it claimed: not because the tier was asserted and applied, but because Tencent's documented default IS the top of a two-position switch. That is a fact about Tencent's serving, not about this run - which is the whole distinction the correction makes.", "fixed_bugs": [ 13, 16, 20, 26, 27, 29, 43, 44, 45, 50, 60, 70, 71, 76, 83, 92, 98, 99 ], "turns": 257, "turns_repo1": 101, "turns_repo2": 156, "group_rank": 6 }, { "id": "GPT-5.6 Terra (high effort)", "n_runs": 1, "model": "GPT-5.6 Terra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-08-27", "color": "#69BFA6", "fixed": 18, "repo1_fixed": 7, "repo2_fixed": 11, "partial": 1, "claimed_only": 0, "extras": 17, "false_positive_fixes": 0, "wall_min": 28.1, "cost_usd": 5.89, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 17, 20, 29, 30, 43, 47, 49, 50, 62, 76, 77, 83, 92, 98, 99, 102 ], "turns": 128, "turns_repo1": 71, "turns_repo2": 57, "group_rank": 502 }, { "id": "Sonnet 5.5 (medium effort)", "n_runs": 1, "model": "Sonnet 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-09-28", "color": "#E68F74", "fixed": 18, "repo1_fixed": 8, "repo2_fixed": 10, "partial": 2, "claimed_only": 4, "extras": 8, "false_positive_fixes": 0, "wall_min": 11.9, "cost_usd": 8.39, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-30 by the mean of 3 (23.3 of 105). This row reported 18 of 105 from a single run. The replicates are byte-identical re-runs; across 3 the runs span 18 to 28. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra runs.", "caveat": "ONE RUN, not a mean - this board has measured a 13-point spread between two runs of a single identical configuration, so treat this as one draw. Both Sonnet 5 rows here compacted mid-task on older CLI builds, so the gap against them is not a clean version comparison. Cost is a list-rate estimate at Sonnet 5.5's published rates; the harness priced this model on another model's table, so the check is on token counts, not on its dollar figure. The machine was idle for every leg.", "note": "Anthropic's Sonnet 5.5 at medium effort through Claude Code on the first-party API, measured the evening it shipped. 18 of 105: 8 of 45 on repo 1 and 10 of 60 on repo 2, 8 genuine extras, 11.9 minutes, $8.39. ONE RUN. For comparison the board currently publishes Sonnet 5 at max effort 9, Sonnet 5 at high effort 9, Opus 5.5 at max effort 41.7 (mean of 3) of 105. *** HOW IT SPENT THE RUN *** On repo 1 it took 43 turns (the harness's own count); it first wrote its report 5 minutes in and its last model call came at minute 5. For scale: Opus 5.5 at max took 159-201 turns there over three runs of 26-37 minutes; Sonnet 5 at max took 207, and every one of those runs wrote its report within 1 minute of its last call. On repo 2 it took 69 turns (the harness's own count); it first wrote its report 7 minutes in and its last model call came at minute 7. For scale: Opus 5.5 at max took 264-317 turns there over three runs of 34-39 minutes; Sonnet 5 at max took 204, and every one of those runs wrote its report within 1 minute of its last call. *** SAME MACHINE AS THE OPUS 5.5 LADDER *** configured byte-identical to the opus55* arms except the model slug and the effort flag: Claude Code straight onto Anthropic's first-party API, no proxy, 1M window requested with the [1m] suffix, CLI-default autocompact, same prompt, same timeout class, same blind judge. The two Sonnet 5 rows on this board ran older Claude Code builds (2.1.218, 2.1.267) and compacted mid-task - 5 compactions across their 4 legs, peaks 284,197 to 295,210 tokens - so a gap against them mixes the model version with how much of each run's context survived, and is not a clean version delta. Effort is a first-party Claude Code tier (documented enum low|medium|high|xhigh|max, enforced: an invented tier is answered 'Unknown --effort value'), taken at face value like every other Claude row here. The harness stated init.model claude-sonnet-5-5[1m] on every leg - the [1m] suffix honoured, so this is a 1M-window run - on Claude Code 2.1.283. Peak context 397,488 tokens; 0 compactions across 2 legs. Autocompact ran at the CLI default. *** COST IS A LIST ESTIMATE *** token counts at Anthropic's published Sonnet 5.5 rates (as read 2026-09-28): $2 input, $10 output, $0.20 cache read, $4 cache write - the 1-hour rate, which is what this harness writes. The harness's own cost figure is NOT a cross-check on this model: Claude Code reported its cost basis for Sonnet 5.5 as 'unknown' and priced the run on Opus 5.5's table - its figure is exactly those tokens at Opus 5.5 rates, about a fifth higher than Sonnet 5.5's own. So the check here is on tokens: each leg's metered counts agree with the harness's own per-model counts within 3 per cent. Claude Code on a subscription exposes no invoice, hence `list`. *** RECEIPTS *** All legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. Judge: blind Codex gpt-5.5 - an OpenAI model scoring an Anthropic one, so neither the model itself nor a sibling of it. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 13, 20, 22, 29, 30, 39, 43, 45, 50, 71, 76, 77, 83, 92, 97, 98, 99, 100 ], "turns": 112, "turns_repo1": 43, "turns_repo2": 69, "group_rank": 503 }, { "id": "Gemini 3.7 Flash (high; Antigravity CLI retest)", "n_runs": 1, "model": "Gemini 3.7 Flash", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-08-25", "color": "#6697E9", "fixed": 18, "repo1_fixed": 4, "repo2_fixed": 14, "partial": 3, "claimed_only": 0, "extras": 11, "false_positive_fixes": 0, "wall_min": 37.5, "cost_usd": 8.57, "cost_kind": "list", "wall_note": "Both repo legs ran concurrently on one machine and one key; contention can inflate this figure, not shrink it.", "ceiling": true, "featured": false, "superseded": "Sequential re-run on 2026-08-28 (16/105) supersedes this concurrent-legs run for a comparable wall.", "caveat": null, "note": "Google's current CLI.", "fixed_bugs": [ 16, 20, 43, 45, 47, 49, 50, 60, 62, 66, 76, 83, 84, 92, 97, 98, 99, 100 ], "turns": 450, "turns_repo1": 138, "turns_repo2": 312, "group_rank": 202 }, { "id": "Muse Spark 1.3 (high effort, Muse Code, OpenRouter)", "n_runs": 1, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / OpenRouter", "effort": "high", "effort_status": "verified", "date": "2026-09-12", "color": "#E57EA2", "fixed": 18, "repo1_fixed": 6, "repo2_fixed": 12, "partial": 1, "claimed_only": 0, "extras": 17, "false_positive_fixes": 0, "wall_min": 53.3, "cost_usd": 9.53, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded by the Meta first-party row at the SAME TIER (19/105). Both ran Muse Code at high on Muse Spark 1.3; this one reached the model through OpenRouter and that one through api.meta.ai directly, so the pair is a clean route comparison and the first-party leg is the one to quote. The route is worth about a point - 18 against 19 - which is the useful thing this row establishes on its way out. It is kept rather than deleted because it is the only REAL BILL on a Muse Code row: the first-party rows run on a subscription account that exposes no per-run credits delta, so their cost columns are list estimates and this one is not.", "caveat": "One run, at high. THE ORIGINAL CAVEAT HERE CLAIMED THE OPPOSITE AND IS RETRACTED: it said the aggregator hop was PROVEN inert, that Muse Code sends high and OpenRouter drops it, and that four receipts at n=9 showed every tier overlapping with bogus_zzz returning HTTP 200. Every one of those receipts probed api/v1/MESSAGES with a flat top-level reasoning_effort, which is not the surface this harness uses; on api/v1/RESPONSES, where the shim actually forwards, the tiers separate, an invented tier is refused 400, and the field is applied rather than dropped. Muse Code does send high - that part was right - and the route honours it, which is why this row now reads high/verified. The 402 that forced this row onto an aggregator is unrelated and still true. The route is an aggregator because Meta's own endpoint answers 402 here, billing verification failed, which is also why cost is a REAL OpenRouter credits delta rather than a list estimate. The benchmark repos were re-installed from the same lockfile inside Linux before the agent started, because the Windows install lays down win32 binaries Linux node cannot execute and an agent that cannot run the suite is solving a different task than every other row.", "note": "The first row on this board driven by a vendor's OWN agent rather than by Claude Code. *** ITS EFFORT LABEL WAS WRONG WHEN PUBLISHED AND IS CORRECTED HERE (2026-09-13) *** It shipped as default/inert_default on the strength of four receipts showing every tier overlapping on OpenRouter. Those receipts probed api/v1/MESSAGES with a flat top-level reasoning_effort. Muse Code speaks the RESPONSES API and the shim forwards to api/v1/RESPONSES, touching only the model id, the provider pin and the stream flag. Re-probed on THAT surface with the nested reasoning:{effort} the harness actually sends, at n=3 per tier on the same prompt as the Meta first-party probe, the dial is not inert: mean reasoning tokens run 845, 2644, 5111, 7858, 10114 and 13243 across minimal to max - a 15.7x spread against a 1.46x worst within-tier spread, with minimal, low, medium and high mutually disjoint - an invented tier is refused HTTP 400, and an omitted field echoes medium. AND THE ARM NEVER USED THAT DEFAULT: captured at the request body, an unflagged Muse Code sends reasoning.effort=high on its main loop rather than omitting the field, so this row ran at high and is relabelled high/verified. The score, cost, wall and extras are unchanged - only the label was ever wrong. Receipt: effort-dial-probes/20260913-musespark13-openrouter-responses-magnitude.txt, and the four superseded receipts carry a correction banner rather than being deleted, because their finding stands for the surface they measured. Muse Code 0.1.0 runs the loop, picks the tools and decides when to stop, and Meta's model answers it - the same experiment the Kimi CLI row runs for Moonshot. *** DO NOT READ THIS AS A HARNESS A/B AGAINST THE OTHER META ROW *** The board's only other Meta row is Muse Spark 1.2 in Claude Code, on Meta's own endpoint, at a genuine xhigh (17/105). Against this row the MODEL VERSION, the HARNESS, the ROUTE and the EFFORT all differ, so the pair is a comparison of two stacks as shipped and not an isolation of any one of them. Muse Code is chattier than a single agent loop by design: it runs a reminder-observer side-agent alongside the main one, so the turn count and the token bill both include work no Claude Code row does. *** WHY THIS ROW NEEDED TWO HARNESS FIXES BEFORE IT COULD EXIST *** Neither is a property of the model and both are disclosed because they changed what ran. First, the launcher that starts the Meta-API shim inside WSL had never once worked - it backgrounded the shim inside a session that is torn down the instant the launching shell exits, so the shim died before it could create its own log file. Second, and less visibly, the shim was not asking upstream for a stream: Muse Code requests one with an Accept header, OpenRouter reads the body, so every turn arrived as a single blob when generation finished - and Muse Code kills a run whose model stream has been idle for 180 seconds, on a route that routinely takes 200 to 230 seconds per call. The first attempt at this arm was billed for a response it never saw. Both legs of this row ran with both fixes in place.", "fixed_bugs": [ 20, 25, 30, 41, 43, 45, 49, 50, 71, 76, 77, 84, 95, 96, 98, 99, 100, 102 ], "turns": 235, "turns_repo1": 109, "turns_repo2": 126, "group_rank": 202 }, { "id": "Gemini 3.8 Flash - mean of 3", "n_runs": 3, "model": "Gemini 3.8 Flash", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-09-15", "color": "#4983E4", "fixed": 18, "repo1_fixed": 6.7, "repo2_fixed": 11.3, "partial": 1.3, "claimed_only": 0.3, "extras": 4, "false_positive_fixes": 0, "wall_min": 38.7, "cost_usd": 11.03, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": true, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 15 to 20. This row SUPERSEDES the single-run row (20/105) published 2026-09-02, which was the HIGHEST of the three; the supersession is about method, not a correction. Every figure here is a MEAN carried to one decimal - 18.0 of 105. THE REPLICATION IS NOT VERIFIED: the Antigravity CLI writes no init record, so neither the context regime nor the harness version is recoverable for any of the three runs. AND THE WALL COLUMN IS NOT COMPARABLE TO ITS CONTROL: all four replicate legs ran 17 to 32 per cent slower per token than the 2026-09-02 run, and the cause is not established - box load and provider throttling were both tested and both failed to explain it.", "note": "Google's newest Flash on its own agent CLI, RUN THREE TIMES. Runs scored 20, 19 and 15 of 105 (repo 1: 7, 7, 6 - repo 2: 13, 12, 9). Mean 18.0, range 5. *** THE REPLICATION IS UNVERIFIED, AND THIS HARNESS IS THE BLINDEST ON THE BOARD *** The Antigravity CLI writes no session init record at all, so nothing about context size, compaction or even its own version is recoverable from the transcripts. Where the Z.ai rows are missing token counts, this one is missing the record that would carry them. So it is not known whether the three runs met the same conditions, and it is not known what harness version the 2026-09-02 control ran against the 2026-09-15 replicates. Stated plainly rather than left to be inferred from an absence. *** THE WALL COLUMN IS SLOWER THAN ITS CONTROL AND NOBODY KNOWS WHY *** All four replicate legs ran at 68, 71, 74 and 83 per cent of the 2026-09-02 control's tokens per minute - 17 to 32 per cent slower, every leg, in one direction. TWO EXPLANATIONS WERE CHECKED AND BOTH FAILED. Box contention: the GLM arm ran under the identical three-lane load at 100 and 143 per cent of its own control, so a busy machine does not explain it. Provider throttling: one leg did die on a hard Google quota wall (Individual quota reached, resets in 3h34m11s) and was re-run, and the throttling story predicted that the re-run - alone on an idle box with a freshly reset quota, the cleanest conditions any leg in this wave got - would come back near the control's rate. It came back the SLOWEST OF THE FOUR. A hypothesis that predicts fast and gets the slowest run of the set under its own best conditions is refuted, not refined. THE CAUSE IS NOT ESTABLISHED, and no cause is asserted on this row. What it does not touch: the scores. Every score on this board is judged from the diff against a withheld answer key, so wall and cost are the only columns a slow leg can move. *** WHAT THE SCORES SHOW *** The published single run scored 20, the TOP of the three. Repo 2 carries almost all of this arm's score and all of its movement (13, 12, 9 against repo 1's 7, 7, 6), and repo 1 produced ZERO genuine unplanted extras on all three runs. *** EVER VS ALWAYS *** Across the three runs this arm ever fixed 24 distinct bugs of 105 against a mean of 18.0, and 14 of those 24 fell in ALL THREE runs - 2 in two, 8 in exactly one. It fixes the FEWEST distinct bugs of the three flash arms replicated in this wave and repeats the largest share of them. Counts over three runs, not a measured reliability. *** RECEIPTS *** All six scored legs exited 0; the quota-killed leg scored nothing and is not in this mean. Per-leg (repo 1 / repo 2): wall 13.2/16.6, 20.7/22.9 and 18.6/24.2 minutes; cost 5.0131/4.7741, 6.4788/5.3301 and 5.9592/5.5222 dollars. Genuine extras 6, 2 and 4. Claimed-only 1, 0 and 0 across six legs. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible. high is the TOP tier this CLI offers for 3.8 Flash - agy models lists only low, medium and high, and --effort max is rejected outright, so it cannot silently clamp. billing=LIST ESTIMATE, NOT A BILL: Google list rates through 2026-12-31. Judge: Codex gpt-5.5 on all six legs.", "fixed_bugs": [ 13, 16, 20, 27, 30, 33, 34, 45, 47, 50, 60, 62, 71, 76, 80, 82, 83, 84, 92, 97, 98, 99, 100, 102 ], "bug_hits": { "13": 1, "16": 3, "20": 3, "27": 1, "30": 3, "33": 3, "34": 3, "45": 3, "47": 3, "50": 1, "60": 3, "62": 1, "71": 2, "76": 3, "80": 1, "82": 1, "83": 3, "84": 3, "92": 3, "97": 3, "98": 1, "99": 3, "100": 2, "102": 1 }, "coverage_runs": 3, "turns": 658, "turns_repo1": 329, "turns_repo2": 329, "group_rank": 102 }, { "id": "GLM-5.3 Flash (max effort, Z.ai API) - mean of 3", "n_runs": 3, "model": "GLM-5.3 Flash", "vendor": "Z.ai", "local": false, "harness": "Claude Code / Z.ai API", "effort": "max", "effort_status": "verified_ceiling", "date": "2026-09-15", "color": "#55C9D8", "fixed": 17.7, "repo1_fixed": 9.3, "repo2_fixed": 8.3, "partial": 1, "claimed_only": 1.3, "extras": 3.3, "false_positive_fixes": 0, "wall_min": 63.2, "cost_usd": 1.03, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 16 to 19. This row SUPERSEDES the single-run max row (19/105) published 2026-09-12, which was the HIGHEST of the three; the supersession is about method, not a correction. Every figure here is a MEAN carried to one decimal - 17.7 of 105. THE REPLICATION IS NOT VERIFIED: Z.ai's endpoint reports no token accounting, so whether the three runs met the same context conditions cannot be checked at all. That is unknown, not fine. The CLI version also differed (2.1.269 for the first run, 2.1.270 for the other two).", "note": "GLM-5.3 Flash's top rung on Z.ai's own endpoint, RUN THREE TIMES. Runs scored 19, 18 and 16 of 105 (repo 1: 10, 9, 9 - repo 2: 9, 9, 7). Mean 17.7, range 3 - the TIGHTEST spread of the three flash arms replicated in this wave. *** THE REPLICATION IS UNVERIFIED, AND THAT IS A PROPERTY OF THE ROUTE *** This board now checks, before publishing a mean, that its members ran in the same harness configuration. On this arm that check CANNOT RUN. Z.ai's Anthropic-compatible endpoint reports input_tokens: 0 on every turn and emits no compaction records, so peak context and compaction count are BLANK, not zero. What is readable: all three runs resolved glm-5.3-flash with the 1M window, and the CLI went 2.1.269 for the first run to 2.1.270 for the other two. Beyond that there is nothing to compare. Read this row as three runs that are believed to match and were not confirmed to - which is a weaker statement than the DeepSeek Flash row on this same board can make, and the difference is worth seeing. *** WHAT THE SPREAD SUPPORTS *** A fixed configuration produced 19, 18 and 16. The published single run scored 19, the TOP of that range. A 3-point range is narrower than the gap between this row and several of its neighbours, but those neighbours are single runs with unmeasured spreads, so no ordering against them is established by this row either. *** AND IT IS THE ARM THAT KILLED THE CONCURRENCY EXCUSE *** Three arms ran in parallel on these nights, which no control had, so the obvious worry was that a busy box slowed the replicates. On this arm it did not: the four replicate legs ran 100, 143, 87 and 106 per cent of the control's tokens per minute under exactly that load. Concurrency is not a uniform tax, and it cannot be written on the other arms' rows as though it explained them. *** EVER VS ALWAYS *** Across the three runs this arm ever fixed 23 distinct bugs of 105 against a mean of 17.7, and 12 of those 23 fell in ALL THREE runs - 6 in two, 5 in exactly one. That is the tightest ever-to-always ratio of the three flash arms replicated in this wave, alongside its tightest score range, and both are counts over three runs rather than a measured reliability. *** RECEIPTS *** All six legs exited 0. Per-leg (repo 1 / repo 2): wall 30.0/28.4, 38.8/32.4 and 33.8/26.3 minutes; cost 0.3225/0.6246, 0.4069/0.6215 and 0.4936/0.6121 dollars. Genuine extras 4, 3 and 3. Claimed-only 1, 3 and 0 across six legs. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible. billing=LIST ESTIMATE, NOT A BILL: Z.ai publishes no balance or usage endpoint on this plan key, so there is no credits delta to read; costed at the same published Flash rates as the other Flash rows, which is what keeps them comparable. Judge: Codex gpt-5.5 on all six legs.", "fixed_bugs": [ 13, 16, 20, 23, 25, 27, 29, 30, 39, 40, 43, 44, 50, 60, 70, 71, 76, 83, 84, 92, 97, 98, 99 ], "bug_hits": { "13": 3, "16": 2, "20": 3, "23": 3, "25": 2, "27": 1, "29": 2, "30": 2, "39": 3, "40": 1, "43": 3, "44": 3, "50": 3, "60": 1, "70": 1, "71": 3, "76": 3, "83": 3, "84": 1, "92": 3, "97": 2, "98": 2, "99": 3 }, "coverage_runs": 3, "turns": 332, "turns_repo1": 119, "turns_repo2": 213, "group_rank": 100 }, { "id": "GPT-6 Luna (max effort)", "n_runs": 1, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "max", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 17, "repo1_fixed": 6, "repo2_fixed": 11, "partial": 1, "claimed_only": 0, "extras": 22, "false_positive_fixes": 0, "wall_min": 85.8, "cost_usd": 0.52, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Superseded 2026-09-23 by the mean of 2 (19.0 of 105). This row reported 17 of 105 from a single run, published the day the arm was added. The replicates are byte-identical re-runs; across 2 the runs span 17 to 21. Nothing here is retracted - it is kept as the receipt for what the board showed before the extra run.", "caveat": "ONE RUN, not a mean - and this board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw. Two replicates are queued and will supersede it. Runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it compares cleanly to the GPT-6 Sol rows, which run the same build. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs.", "note": "OpenAI's GPT-6 Luna at max effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 17 of 105: 6 of 45 on repo 1 and 11 of 60 on repo 2, 22 genuine extras, 85.8 minutes, $0.52. ONE RUN so far; two replicates are queued and will supersede this row. *** THIS IS THE CHEAP END OF THE GPT-6 FAMILY *** list price is $0.10 in and $0.50 out per million, TWENTY TIMES cheaper than GPT-6 Sol ($2 / $10) on both sides of the meter. So the question this row answers is not whether it wins - it is how much of the family's score survives a 20x price cut. For the comparison the board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max, GPT-6 Sol at 29.3 (mean of 3), and this model's own direct predecessor GPT-5.6 Luna at 31.3 (mean of 3). *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration. Read this figure, and any gap between it and a peer, with that width in mind until the replicates land. *** EXISTENCE WAS PROVED POSITIVELY, NOT INFERRED FROM A REJECTION *** the slug gpt-6-luna appears in the account's own model list and the pinned CLI accepted it and answered. That distinction matters on this board: the CLI returns 'The model is not supported when using Codex with a ChatGPT account' word for word for a deliberately fake slug, so a rejection is evidence about the client and never about whether a model exists. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows. It DOES compare cleanly to the GPT-6 Sol rows, which run the same pinned 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 25, 29, 30, 39, 41, 43, 47, 49, 50, 62, 76, 77, 80, 82, 83, 84, 92 ], "turns": 315, "turns_repo1": 210, "turns_repo2": 105, "group_rank": 300 }, { "id": "Grok 4.5 (re-run Aug 6)", "n_runs": 1, "model": "Grok 4.5", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-08-06", "color": "#AAADB1", "fixed": 17, "repo1_fixed": 5, "repo2_fixed": 12, "partial": 0, "claimed_only": 1, "extras": 10, "false_positive_fixes": 0, "wall_min": 27.9, "cost_usd": 8.5, "cost_kind": "floor", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 30, 37, 43, 47, 49, 50, 62, 76, 77, 83, 84, 95, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 202 }, { "id": "Muse Spark 1.2 (xhigh, Meta API)", "n_runs": 1, "model": "Muse Spark 1.2", "vendor": "Meta", "local": false, "harness": "Claude Code / Meta API", "effort": "xhigh", "effort_status": "verified_ceiling", "date": "2026-08-06", "color": "#EDA3BD", "fixed": 17, "repo1_fixed": 6, "repo2_fixed": 11, "partial": 0, "claimed_only": 0, "extras": 12, "false_positive_fixes": 0, "wall_min": 35.7, "cost_usd": 13.99, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "Mixed serving path: repo 1 ran Meta-direct at verified xhigh, repo 2 through OpenRouter at default effort. The single-path number for this model is the 14/105 row.", "note": null, "fixed_bugs": [ 20, 25, 30, 41, 43, 45, 62, 71, 76, 77, 83, 84, 92, 97, 98, 99, 102 ], "turns": 31, "turns_repo1": 1, "turns_repo2": 30, "group_rank": 301 }, { "id": "Ox Alpha (stealth; free)", "n_runs": 1, "model": "Ox Alpha (stealth)", "vendor": "Z.ai", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-25", "color": "#95DEE7", "fixed": 16, "repo1_fixed": 8, "repo2_fixed": 8, "partial": 1, "claimed_only": 1, "extras": 3, "false_positive_fixes": 0, "wall_min": 59.4, "cost_usd": 0.0, "cost_kind": "free", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "OpenRouter stealth slug, free at the time of the run; Z.ai confirmed on 26 Aug 2026 that it was GLM-5.3-Flash. Off the default view because the named, priced run is the GLM-5.3 Flash row. Fixed H3 - a repo-1 bug that had survived every model in every prior run.", "fixed_bugs": [ 16, 20, 22, 25, 27, 32, 39, 43, 50, 71, 76, 83, 84, 92, 97, 99 ], "turns": 263, "turns_repo1": 118, "turns_repo2": 145, "group_rank": 206 }, { "id": "GLM-5.3 Flash (high effort, Z.ai API)", "n_runs": 1, "model": "GLM-5.3 Flash", "vendor": "Z.ai", "local": false, "harness": "Claude Code / Z.ai API", "effort": "high", "effort_status": "verified", "date": "2026-09-13", "color": "#55C9D8", "fixed": 16, "repo1_fixed": 9, "repo2_fixed": 7, "partial": 0, "claimed_only": 2, "extras": 4, "false_positive_fixes": 0, "wall_min": 62.3, "cost_usd": 0.94, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "One run at each of the three tiers, so every step on this dial is n=1 against n=1. The monotonic ordering across three rungs is more than any single pair would give, but it is three single runs and a repeat at any rung could reorder the middle. THE 3-POINT GAP TO max IS NOT CLAIMED AS REAL. It is smaller than the 6-point OpenRouter gap this board declined to credit on the max row, and it should be read the same way: high and max are close enough that this pair does not separate them. What the dial DOES separate is low from the other two, by 7 and 10 points. WALL CLOCK ON THIS ROW IS NOT A CLEAN-ROOM FIGURE AND THE AUTOMATIC MARKER DID NOT CATCH IT. The repo 2 leg overlapped a Meta arm on a different provider for about 3 of its 32 minutes, before that arm was stopped. *** THIS SENTENCE GAVE THE WRONG REASON AND IS CORRECTED 2026-09-13, SAME DAY *** The runner does NOT only snapshot at the start: it samples the live-leg registry at leg start AND at leg end and unions the two, and has since cross-provider parallelism was first allowed. A leg joined later IS caught if it is still running at the end. What actually happened here is narrower: the Meta arm both ARRIVED and DEPARTED between those two samples - it joined about 3 minutes in and was force-killed before this leg finished - so it existed at neither endpoint. Two point samples cannot see an interval that fits strictly between them. Superseded wording, kept verbatim rather than deleted: the runner stamps concurrent_with= by snapshotting the live-leg registry when a leg STARTS, so a leg that is JOINED later records nothing - this one started first and carries no marker. Three minutes of a mostly-installing neighbour is a negligible effect on a 62-minute total and it can only inflate, never shrink, but the row says it rather than implying a measurement it did not make. CLOSED 2026-09-13, after the wave: the runner now samples the peer registry on every tick of the 30-second keep-alive sweep it already runs, and the row carries the union across all samples - so a peer that lives longer than one tick can no longer slip between the endpoints. Verified against a transient peer that is invisible to both. billing=LIST ESTIMATE, NOT A BILL: Z.ai publishes no balance or usage endpoint on this plan key, so no credits delta exists to read; costed at the same published Flash rates the other three Flash rows were costed at, which keeps them comparable - and those rates carry a launch discount that can lapse. This endpoint does not report reasoning tokens separately, so output volume is the only work proxy available across the dial. ctx 1M.", "note": "The MIDDLE rung of Z.ai's three-tier dial, and the row that completes it. low, high and max are the entire accept-list for this family - none, minimal, medium, xhigh and an invented value are all refused HTTP 400 code 1210 - so EVERY TIER THIS ENDPOINT WILL SERVE FOR THIS MODEL NOW HAS A BENCH ROW, which no other model on this board can say. *** THE DIAL IS MONOTONIC ON SCORE *** 9 at low, 16 here, 19 at max, with output volume rising alongside it - 36,777 then 97,332 then 115,653 tokens. This is a dial that works, measured end to end on one serving path with the route, the harness, the model, the prompt and the repos all held fixed. *** AND IT IS NOT MONOTONIC ON VALUE, WHICH IS THE MORE USEFUL HALF *** Cost per strict fix runs 4.6 cents at low, 5.9 cents HERE, and 5.0 cents at max. THE MIDDLE RUNG IS THE WORST BUY ON THE DIAL. Muse Spark 1.3 did the same thing hours earlier on Meta's stack, where xhigh cost 78 cents per fix against 57 at high and 60 at max. Two vendors, two unrelated dials, two different harnesses, same shape - the middle is where the money goes and the score does not follow. That is not something a tier name or a token sweep would tell you. *** ON THIS STACK high IS DOMINATED BY max, ON ALL THREE COLUMNS AT ONCE *** high costs 0.94 dollars against max's 0.95, takes LONGER in wall clock - 62.3 minutes against 58.3 - and scores three fewer. There is no reading of those numbers on which a buyer should pick high here, which is worth saying plainly because the tier name suggests otherwise. *** IT ALSO LOCATES THE OPENROUTER FLASH ROW, WITHOUT SETTLING IT *** That row scores 13 with 70,628 output tokens. Both figures sit BETWEEN low and high on this dial, not at the top of it - so whatever OpenRouter serves for this model, it is demonstrably not the ceiling, even though Z.ai documents max as this family's default. What tier it IS serving remains unknown: that route ignores the effort field this board sends, which is what inert_default records, and nothing here identifies what it substitutes. Honesty profile: 9 of 45 on repo 1 and 7 of 60 on repo 2, no partials on either, ONE claimed-only per repo, and 4 genuine unplanted extras. Two claimed-only sits between the max row's one and the OpenRouter row's five, so the claim record does not track the tier.", "fixed_bugs": [ 16, 20, 21, 25, 27, 28, 30, 39, 43, 50, 60, 66, 71, 76, 83, 99 ], "turns": 344, "turns_repo1": 115, "turns_repo2": 229, "group_rank": 102 }, { "id": "DeepSeek V4-Pro (max effort, DeepSeek API)", "n_runs": 1, "model": "DeepSeek V4-Pro", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / DeepSeek API", "effort": "max", "effort_status": "first_party", "date": "2026-09-10", "color": "#8773D7", "fixed": 16, "repo1_fixed": 8, "repo2_fixed": 8, "partial": 1, "claimed_only": 2, "extras": 4, "false_positive_fixes": 0, "wall_min": 37.2, "cost_usd": 1.89, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": "Two honest limits on this row. (1) The effort label is `first_party`, not `verified`: a high-vs-max thinking sweep on Pro itself came out INCONCLUSIVE at n=3 - means 51.1K vs 85.1K characters, but between-level 1.66x against a worst within-level spread of 1.78x, with the ranges overlapping. The same sweep on V4.1 Flash the same hour was cleanly disjoint, which is why the Flash rows carry `verified` and this one does not. The path-level facts still hold: the endpoint 400-rejects an invented tier, and the served model id was read back as V4-Pro on every gate call. (2) The $1.89 is a real credits delta for the ARM; the per-repo split is that bill allocated by each leg's token value at list, because a runner bug read the wrong meter for the individual legs. Both metrics rows say so.", "note": "The board's first look at DeepSeek's Pro model with an effort dial in front of it. The existing V4-Pro row (Aug 1) ran through OpenRouter at an inert default; this one runs on DeepSeek's own Anthropic-compatible endpoint at an explicitly requested max. It scores 16/105 - better than that row's 10, and well under the V4.1 Flash rows that cost a fraction of it. On this bench the newer Flash generation beats the older Pro flagship at every tier tried: Flash max 24 and Flash high 19 against Pro max 16, at $1.08 and $0.31 against $1.89. Pro is also the slower model here, 37.2 min against Flash max's 42.6 only because it wrote less.", "fixed_bugs": [ 20, 23, 25, 27, 29, 30, 39, 43, 71, 76, 83, 84, 92, 97, 98, 99 ], "turns": 354, "turns_repo1": 123, "turns_repo2": 231, "group_rank": 100 }, { "id": "Grok 4.5", "n_runs": 1, "model": "Grok 4.5", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-07-26", "color": "#AAADB1", "fixed": 16, "repo1_fixed": 5, "repo2_fixed": 11, "partial": 1, "claimed_only": 3, "extras": 7, "false_positive_fixes": 0, "wall_min": 24.9, "cost_usd": 2.58, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": "One of three runs at the same setting (16 / 13 / 17) - the spread is this board's variance evidence.", "caveat": null, "note": null, "fixed_bugs": [ 16, 20, 30, 37, 40, 47, 50, 60, 62, 71, 76, 83, 84, 99, 100, 102 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 202 }, { "id": "Haiku 5.5 (high effort) - mean of 2", "n_runs": 2, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-10-07", "color": "#EBA691", "fixed": 16, "repo1_fixed": 5, "repo2_fixed": 11, "partial": 1.5, "claimed_only": 1.5, "extras": 4, "false_positive_fixes": 0, "wall_min": 19.3, "cost_usd": 3.56, "cost_kind": "list", "wall_note": "Every run took its two repo legs at the same time on one account; contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": null, "caveat": "Mean of two runs of one configuration (15 and 17 of 105), each with both repo legs at once on one account. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at high effort through Claude Code on the first-party API, released 2026-10-07. MEAN OF TWO RUNS of one configuration: 15 and 17 of 105 (repo 1: 5 and 5 of 45 - repo 2: 10 and 12 of 60), mean 16. Claimed-only fixes 1 and 2; genuine unplanted extras 4 and 4. Walls 17.9 and 20.7 minutes; cost $3.30 and $3.82. Effort high was not captured on the wire separately; the xhigh and max captures on the same CLI build showed the flag passed through verbatim for this model. A first-party Claude Code tier, accepted, not magnitude-probed; Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** EVERY RUN TOOK ITS TWO REPO LEGS AT THE SAME TIME *** on one account, so walls include contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and in every run each leg's priced responses were checked to be exactly the ones in its own stream, with none shared between the legs. *** COST, PRICED PER REQUEST *** 316 of 338 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 20, 25, 27, 30, 43, 45, 47, 49, 62, 71, 76, 77, 82, 83, 84, 92, 97, 98, 99, 100 ], "bug_hits": { "20": 1, "25": 2, "27": 1, "30": 2, "43": 2, "45": 2, "47": 1, "49": 2, "62": 1, "71": 2, "76": 2, "77": 1, "82": 1, "83": 2, "84": 2, "92": 1, "97": 2, "98": 1, "99": 2, "100": 2 }, "coverage_runs": 2, "turns": 216, "turns_repo1": 67, "turns_repo2": 149, "group_rank": 702 }, { "id": "Gemini 3.7 Flash seq", "n_runs": 1, "model": "Gemini 3.7 Flash", "vendor": "Google", "local": false, "harness": "Antigravity CLI", "effort": "high", "effort_status": "verified_ceiling", "date": "2026-08-28", "color": "#6697E9", "fixed": 16, "repo1_fixed": 4, "repo2_fixed": 12, "partial": 1, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 22.7, "cost_usd": 6.36, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "Sequential-legs re-run of the 08-25 concurrent Antigravity run (18/105 vs this 16/105 - n=2 agreement); comparable single-machine wall, so it replaces the dotted original.", "fixed_bugs": [ 20, 30, 43, 45, 47, 50, 60, 71, 76, 82, 83, 92, 97, 98, 99, 102 ], "turns": 363, "turns_repo1": 136, "turns_repo2": 227, "group_rank": 202 }, { "id": "Grok 4.7 (low) - mean of 3", "n_runs": 3, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "low", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 15.7, "repo1_fixed": 10.3, "repo2_fixed": 5.3, "partial": 0, "claimed_only": 0.7, "extras": 18.3, "false_positive_fixes": 0, "wall_min": 22.2, "cost_usd": 9.36, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Three runs of an identical configuration spanning 11 to 21 of 105. The Grok 4.6 row at this rung is a SINGLE run, so that comparison is three draws against one and is not a comparison of two distributions. Cost is a reconstructed FLOOR rather than a bill, and uses a per-session arithmetic the 4.6 rows on this board do not yet use; the note gives both figures. The runs spawned different numbers of subagent sessions on identical arm definitions, which the board's regime check cannot verify on this harness.", "note": "Grok 4.7 at low through xAI's own CLI, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 11, 21 and 15 of 105 (repo 1: 5, 15, 11 of 45 - repo 2: 6, 6, 4 of 60). Mean 15.7, spread 10. This supersedes the single-run row published earlier the same day at 11 of 105, which said the same thing on one draw. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 at low scored 15 of 105 (6 on repo 1, 9 on repo 2) - A SINGLE RUN, so the samples are not the same size and the comparison is three draws against one. Difference +0.7. That 4.6 number falls INSIDE the spread of these three runs, which is the clearest way to say the two are not distinguishable on this evidence. *** WHERE THIS RUNG SITS ON ITS OWN DIAL *** The same model at xhigh is 28.8 of 105 over four runs; this rung is 15.7 over three, a gap of +13.1. The dial as the board now shows it: low 15.7 (n=3), medium 26.7 (n=3), high 19.3 (n=3), xhigh 28.8 (n=4). For comparison Grok 4.6's dial ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, each of those from a single run. Whether that flatness is a property of the dial or a coincidence of two single draws is exactly what replicating these rungs was for. *** WHAT THE SPREAD ITSELF SAYS *** 10 points between the best and worst run of an IDENTICAL configuration - the three arms are clones and differ only in name. Any single-run number at this rung, including the one this row replaces, was carrying that much slack without being able to show it. Read any n=1 row on this board, from any vendor, with that in mind. *** COST AND TIME *** Reconstructed floors of $5.41, $10.88, $11.78 for a mean of $9.36; walls 15.1, 20.1, 31.3 minutes, mean 22.2. COST IS A FLOOR, not a bill: the grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates on the same 2/0.5/6 per MTok basis, and xAI bills a request whose prompt reaches 200K at DOUBLE those rates. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic that the 4.6 rows on this board still use it would read $9.71. Compare cost across rows only where both use the same arithmetic. *** GENUINE EXTRAS *** Real defects found in the repo that are not part of the planted set ran 12, 25 and 18 across the three runs, a mean of 18.3. Extras are not scored; they are reported because a model that finds real bugs nobody planted is doing something the planted-bug total cannot see. *** THE TIER WAS CHECKED, NOT ASSUMED, AND AT THE LOWER RUNGS THAT MATTERS MOST *** A zero-token ACP readback before each of these three launches returned reasoning_effort low for a requested low, with the served model named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except that readback - the run would be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. *** RECEIPTS *** All six legs exited 0. Legs SEQUENTIAL within every run, one repo at a time, and only one grok run on the machine at a time - two grok agents share one login. Sessions spawned per leg (repo 1 / repo 2): run 1 1 and 1; run 2 4 and 2; run 3 3 and 1. That count varies run to run on identical arm definitions for this model, tracks cost closely, and has tracked score not at all. The board's automatic regime check parses Claude Code transcripts, so it returns NOT CHECKED for every grok row here and the session counts above are the manual substitute. Judge: blind Codex gpt-5.5 on all six legs, which is neither this model nor a sibling of it.", "fixed_bugs": [ 13, 16, 20, 21, 22, 23, 25, 29, 30, 37, 39, 41, 43, 44, 45, 49, 60, 76, 83, 84, 95, 99, 100 ], "bug_hits": { "13": 2, "16": 2, "20": 3, "21": 1, "22": 2, "23": 2, "25": 2, "29": 1, "30": 3, "37": 3, "39": 3, "41": 2, "43": 2, "44": 1, "45": 2, "49": 2, "60": 1, "76": 2, "83": 3, "84": 2, "95": 1, "99": 2, "100": 3 }, "coverage_runs": 3, "turns": 285, "turns_repo1": 142, "turns_repo2": 144, "group_rank": 4 }, { "id": "Haiku 5.5 (high effort)", "n_runs": 1, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-10-07", "color": "#EBA691", "fixed": 15, "repo1_fixed": 5, "repo2_fixed": 10, "partial": 1, "claimed_only": 1, "extras": 4, "false_positive_fixes": 0, "wall_min": 17.9, "cost_usd": 3.3, "cost_kind": "list", "wall_note": "Both repo legs ran at the same time on one account; contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": "Superseded 2026-10-07 by the mean of two runs (16 of 105): this row was run 1 alone, at 15; run 2 scored 17. Nothing here is retracted - it is the receipt for what the board showed before run 2.", "caveat": "One run at high effort, both repo legs at once on one account. Single runs on this board move by more than 2 points against a three-run mean about half the time. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at high effort through Claude Code on the first-party API, released 2026-10-07. ONE RUN: 15 of 105 - 5 of 45 on repo 1 and 10 of 60 on repo 2 - with 1 partial fix(es), 1 claimed-only and 4 genuine unplanted extras; 17.9 minutes, $3.30. The same model's xhigh row is a mean of three runs; this is a single run, and on this board a first run has moved by more than 2 points against its own three-run mean about half the time. *** EFFORT: HIGH *** a first-party Claude Code tier. Not captured on the wire for this run: the xhigh and max captures on the same CLI build showed the flag passed through verbatim for this model, and high goes through the same path. Accepted, not magnitude-probed. Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** BOTH REPO LEGS RAN AT THE SAME TIME *** on one account, so each leg's wall includes the other's contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and each leg's priced responses were checked to be exactly the ones in its own stream, with none shared between the legs. *** COST, PRICED PER REQUEST *** 158 of 172 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates; 98.3% of prompt tokens were cache reads. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 20, 25, 30, 43, 45, 49, 62, 71, 76, 83, 84, 97, 98, 99, 100 ], "turns": 205, "turns_repo1": 65, "turns_repo2": 140, "group_rank": 702 }, { "id": "GPT-5.6 Terra (medium effort)", "n_runs": 1, "model": "GPT-5.6 Terra", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-08-27", "color": "#69BFA6", "fixed": 15, "repo1_fixed": 4, "repo2_fixed": 11, "partial": 1, "claimed_only": 0, "extras": 6, "false_positive_fixes": 0, "wall_min": 20.2, "cost_usd": 3.87, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 20, 28, 30, 39, 47, 71, 76, 77, 82, 84, 97, 98, 99, 100, 102 ], "turns": 93, "turns_repo1": 48, "turns_repo2": 45, "group_rank": 503 }, { "id": "Grok 4.6 (low)", "n_runs": 1, "model": "Grok 4.6", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "low", "effort_status": "verified", "date": "2026-08-27", "color": "#757A82", "fixed": 15, "repo1_fixed": 6, "repo2_fixed": 9, "partial": 4, "claimed_only": 0, "extras": 9, "false_positive_fixes": 0, "wall_min": 18.0, "cost_usd": 4.14, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 30, 33, 34, 47, 49, 50, 76, 83, 84, 92, 99, 100 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 104 }, { "id": "Qwen3.8-27B (xhigh effort, Alibaba API) - mean of 3", "n_runs": 3, "model": "Qwen3.8-27B", "vendor": "Alibaba", "local": true, "harness": "Claude Code / Alibaba API", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-15", "color": "#ECB149", "fixed": 15, "repo1_fixed": 5.7, "repo2_fixed": 9.3, "partial": 0.7, "claimed_only": 4.3, "extras": 2.3, "false_positive_fixes": 0, "wall_min": 56.8, "cost_usd": 5.55, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Every figure on this row is a MEAN of three runs, carried to one decimal - 15.0 of 105, range 4 (13 to 17). This row SUPERSEDES the single-run row (15/105) published 2026-09-12, which landed exactly on this mean by coincidence of one draw, not because a single run and a mean agree in general - see the range before reading any margin drawn against 15 as precise. effort_status stays first_party rather than verified for the same reason it did at n=1: what is verified is that the field is accepted and validated, not that xhigh buys more thinking than a lower tier on this specific model - the magnitude sweep exists for the other two Qwen3.8 sizes and not for this one. Cost is a list-price ESTIMATE, not a bill, as above. The three runs were verified to have run in the same harness regime on both repos - same model resolved, zero compactions across all six legs, peak live context 26,047-27,567 against a 1,000,000-token declared window - which is the exception on this board rather than the rule. One thing was not held fixed and is on the record: CLI 2.1.268 for the first run, 2.1.272 for the other two.", "note": "THE MIDDLE SIZE OF THE QWEN3.8 FAMILY, RUN THREE TIMES ON ALIBABA'S OWN FIRST-PARTY ENDPOINT. Runs scored 15, 13 and 17 of 105 (repo 1: 6, 5, 6 - repo 2: 9, 8, 11). Mean 15.0, range 4. *** THE FINDING FROM THE SINGLE RUN SURVIVES AT n=3 *** The n=1 row (15/105) was the most interesting Qwen row on the board because it LOST to both siblings and cost more than one of them. The mean does not soften that: at 15.0 it is still eleven points below Qwen3.8-Flash's 26/105 and well below Qwen3.8-Max's 28/105, on the same endpoint, the same harness and the same two repos. It is still the fast one - a combined wall mean of 56.8 minutes against Flash's 97.3 and Max's 103.4 - so read it as a latency option rather than a price/capability midpoint; on capability per dollar there is still no case for it here. *** THE EFFORT LABEL IS MEASURED, NOT REQUESTED, AND THAT DOES NOT CHANGE WITH REPLICATION *** This arm is commissioned at DEFAULT effort. On this route there is no such thing: a logging pass-through of the CLI (receipt: effort-dial-probes/20260912-claude-code-wire-default-effort.txt) shows an unflagged Claude Code putting output_config.effort=xhigh on every main call, and the endpoint validates and echoes that field. The dial itself was probed once, not three times - it is a property of the endpoint's wire protocol, not of an individual run, so it does not need re-verifying per leg. Honesty profile across the three runs, mean per repo: repo 1 fixed 5.7/45 with zero partials, repo 2 fixed 9.3/60 with 0.7 partials; claimed-only 2.0 and 2.3; genuine unplanted extras 0.7 and 1.7; false-positive fixes zero on both, all three runs. Per-leg receipts (repo 1 / repo 2): wall 24.8/22.0, 24.5/43.4 and 26.8/28.8 minutes; cost 2.48/3.61, 1.63/3.60 and 2.11/3.23 dollars. All six legs were clean - every leg exited 0, none carries a handicap note. ONE WALL FIGURE IS FLAGGED, NOT EXPLAINED: two of the three repo-2 legs shared the machine with another running leg, and this board's own metrics record that as contention rather than silently folding it into the mean. Contention can inflate a wall figure, never shrink it, and it does not touch score, tokens or cost. The two contended legs landed nowhere near each other (43.4 and 28.8 minutes) and the one uncontended control (22.0) is the fastest of the three, so the direction is consistent with contention costing time but the SIZE of the effect is not - no cause beyond 'shared the box' is asserted here. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible. billing=LIST ESTIMATE, NOT A BILL: DashScope exposes no credits API on this path, so cost is computed from each run's own token counts at 0.425 in / 0.5312 cache-write / 0.085 cache-read / 2.55 out per MTok - the same rates the n=1 row used, from OpenRouter's Alibaba-endpoint listing for this slug. Cache writes are priced and not zero: Qwen is the one family on this board that is charged for them.", "fixed_bugs": [ 16, 20, 27, 30, 32, 39, 43, 44, 45, 47, 49, 50, 60, 61, 62, 71, 76, 83, 84, 92, 97, 98, 99 ], "bug_hits": { "16": 2, "20": 3, "27": 1, "30": 3, "32": 1, "39": 2, "43": 3, "44": 1, "45": 1, "47": 1, "49": 1, "50": 3, "60": 3, "61": 1, "62": 1, "71": 3, "76": 3, "83": 3, "84": 1, "92": 1, "97": 2, "98": 2, "99": 3 }, "coverage_runs": 3, "turns": 292, "turns_repo1": 101, "turns_repo2": 191, "group_rank": 101 }, { "id": "Qwen3.8-27B (xhigh effort, Alibaba API)", "n_runs": 1, "model": "Qwen3.8-27B", "vendor": "Alibaba", "local": true, "harness": "Claude Code / Alibaba API", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-12", "color": "#ECB149", "fixed": 15, "repo1_fixed": 6, "repo2_fixed": 9, "partial": 0, "claimed_only": 4, "extras": 2, "false_positive_fixes": 0, "wall_min": 46.8, "cost_usd": 6.09, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row at the same rung (15.0/105, range 4). This row is one of three runs that scored 15, 13 and 17, and it landed exactly on the eventual mean - a coincidence of one draw, not a property of the config. Nothing here is retracted: the 15 happened and this is its receipt, and it happens to also be this rung's most-representative single number. Kept rather than deleted because a supersession is only legible next to the row it supersedes.", "caveat": "One run, and the effort_status is first_party rather than verified for a specific reason: the MAGNITUDE of the dial was probed on this endpoint for the other two Qwen3.8 sizes (both bind hard at the bottom and saturate across high/xhigh/max) but not for this one, so what is verified here is that the field is accepted and validated, not that xhigh buys more thinking than high on this particular model. Given the shape the siblings show, expect it to buy little. Cost is a list-price ESTIMATE and not a bill - DashScope exposes no credits API on this path - computed from the run's own token counts at 0.425 in / 0.5312 cache-write / 0.085 cache-read / 2.55 out per MTok, from OpenRouter's Alibaba-endpoint listing for this slug; that source is trusted here because for Qwen3.8-Max it lists exactly the DashScope rates this board already carries. Cache writes are priced at 0.5312 and NOT zero: Qwen is the one family on this board that is charged for them, and modelling them free is the error the Aug 3 Qwen row shipped with. The model id was probed rather than assumed - the endpoint serves `qwen3.8-27b` and 400s on -instruct, -0902 and qwen3-8-27b.", "note": "The middle size of the Qwen3.8 family, on Alibaba's own Anthropic-compatible endpoint - the same first-party path as the Qwen3.8-Max and Qwen3.8-Flash rows, so the three sizes differ by model and by nothing else. It is the most interesting Qwen row on the board because it LOSES to both siblings and costs more than one of them: 15/105 for $6.09 against Flash's 26/105 for $1.81 and Max's 28/105 for $26.29. Eleven points below a model that costs a THIRD as much, on the same endpoint, in the same harness, on the same two repos. It is the fast one - 46.8 minutes against 97.3 and 103.4, less than half the wall clock of either sibling - so read it as a latency option and not a price/capability midpoint, because on capability per dollar there is no case for it here. *** THE EFFORT LABEL IS MEASURED, AND IT IS NOT WHAT WAS ASKED FOR *** This arm was commissioned at DEFAULT effort. On this route there is no such thing. A logging pass-through of the CLI (receipt: effort-dial-probes/20260912-claude-code-wire-default-effort.txt) shows an UNFLAGGED Claude Code putting output_config.effort=xhigh on every main call - Claude Code has no send-nothing mode - and this endpoint IMPLEMENTS that field: it 400-rejects bogus_zzz and names its accept-list (low, medium, high, xhigh, max), the same five the other two Qwen3.8 sizes expose. So an out-of-the-box run here is an xhigh run, the row says xhigh, and the arm passes --effort xhigh explicitly so the result does not depend on a CLI default that can change under it. An unrequested tier is not an absent tier. Honesty profile: 15 strict matches, ZERO partials, 4 claimed-only across both repos, 2 genuine unplanted extras and no false-positive fixes.", "fixed_bugs": [ 16, 20, 30, 39, 43, 45, 50, 60, 62, 71, 76, 83, 84, 92, 99 ], "turns": 318, "turns_repo1": 120, "turns_repo2": 198, "group_rank": 101 }, { "id": "Mistral Large 4 - mean of 3", "n_runs": 3, "model": "Mistral Large 4", "vendor": "Mistral", "local": false, "harness": "Claude Code / OpenRouter", "effort": "high", "effort_status": "first_party", "date": "2026-10-07", "color": "#E10500", "fixed": 15, "repo1_fixed": 5, "repo2_fixed": 10, "partial": 0.7, "claimed_only": 0.7, "extras": 3.3, "false_positive_fixes": 0, "wall_min": 111.1, "cost_usd": 11.95, "cost_kind": "bill", "wall_note": "Run 2 took its two repo legs at the same time, and run 3's repo-1 leg ran alongside a repo-2 attempt that was voided; contention can inflate this figure, not shrink it.", "ceiling": true, "featured": false, "superseded": null, "caveat": "Score is the mean of three runs (17, 11 and 17 of 105); one leg scored zero because the model ended its session early by writing a tool call as text. Time and cost are the mean of the two cache-fixed runs - run 1's were inflated by a prompt-cache failure (its bill about fourfold), kept in its receipts. One attempt that hit the model's context limit was voided as a harness fault and re-run. Prices are a 50% launch discount. Run 1 differs from runs 2 and 3 in one harness detail (where a per-turn token counter sits in the prompt). Effort is high, the only reasoning-on value Mistral documents for this model, sent on every call; through OpenRouter no value measurably changed its output. Three of the six legs ran alongside another leg, so their walls include contention.", "note": "Mistral Large 4 (mistralai/mistral-large-4-0, released 2026-10-06 as an API preview) driven by Claude Code through an Anthropic-API shim onto OpenRouter, pinned to Mistral, the only host. MEAN OF THREE RUNS: 17, 11 and 17 of 105 (repo 1: 7, 0 and 8 of 45 - repo 2: 10, 11 and 9 of 60), mean 15.0. Genuine extras 5, 4 and 1, mean 3.3. *** THE ZERO LEG IS THE MODEL'S *** run 2's repo-1 leg: after 62 turns and 56 successful tool calls it wrote its next tool call as plain text in a code block instead of calling the tool, so the harness took it as a final answer and the session ended - no edits, no report. Mistral's own API ended that turn as a normal stop with text, not a tool call, and the environment was healthy, so by this board's rule it is a real zero, not a void. *** ONE ATTEMPT WAS VOIDED AND RE-RUN *** run 3's first repo-2 attempt reached Mistral's 524,288-token context limit after 93 minutes; the harness's emergency compaction then failed (the summary exceeded the CLI's output cap) and the leg ended without a report. That is a harness fault - Claude Code had been told the window was 1M, so it never compacted early - and the leg was re-run with the identical configuration. Its $12.69 is not in the cost. *** TIME AND COST ARE THE MEAN OF RUNS 2 AND 3 ONLY (111.1 minutes, $11.95) *** run 1 took 151.2 minutes and billed $47.28 because its prompt cache never warmed: 1.1% of 72.5M prompt tokens were cache reads. Claude Code sends system messages inside the conversation (an environment block and a per-turn token counter), and on this host they broke prefix caching, so every turn re-processed the whole conversation - the clock and the bill carry the same fault. Runs 2 and 3 moved those messages, text unchanged, into the user turn before them, and read 79.8% of 122.3M prompt tokens from cache. Runs 2 and 3: 95.4 and 126.8 minutes, $8.59 and $15.30. Only the score is averaged over all three; every run's own figures stay in the per-leg receipts. Prices are Mistral's launch discount (50% off its list), so the cost will roughly double at standard rates. Run 2 ran its two repos at the same time, run 3's repo 1 ran alongside the voided attempt, and run 1's legs and run 3's counted repo-2 leg ran alone. 3 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. *** RUN 1 IS NOT BYTE-IDENTICAL TO RUNS 2 AND 3 *** it differs in where that token counter sits in the prompt (system message vs. user turn), the same text in a different role. Runs 2 and 3 are identical. Averaged on purpose, and said here rather than implied away. *** REPLICATION *** the board's automatic same-regime check cannot read context accounting through this route, so it is unverified there; checked from the runs' own streams instead: no counted leg compacted its context. *** EFFORT: HIGH, WHICH IS ALSO THE CEILING *** The proxy sends reasoning effort `high` on every call. Mistral documents exactly two values for this model, `high` (reasoning on) and `none`, so `high` is its only reasoning-on tier and therefore its top one. What the label rests on is that request and Mistral's own vocabulary, not a separating probe: through OpenRouter, 7 conditions including `none` and an invented value (n=9 each) all overlapped, with reasoning present on every probe call, so this route could not show the field binding. Mistral's own API was not reachable for these runs. *** RECEIPTS *** All six counted legs exited 0. Judge: blind Codex gpt-5.5 on the 5 legs that changed code; the sixth changed no file and wrote no report, so there was nothing to judge and it scores 0 by rule. The judge is neither this model nor a sibling of it.", "fixed_bugs": [ 16, 20, 25, 26, 27, 29, 30, 39, 40, 43, 47, 49, 50, 60, 62, 71, 76, 83, 92, 97, 98, 99, 102 ], "bug_hits": { "16": 1, "20": 2, "25": 2, "26": 1, "27": 1, "29": 2, "30": 1, "39": 2, "40": 1, "43": 2, "47": 2, "49": 1, "50": 3, "60": 2, "62": 2, "71": 3, "76": 3, "83": 3, "92": 1, "97": 3, "98": 3, "99": 3, "102": 1 }, "coverage_runs": 3, "turns": 386, "turns_repo1": 117, "turns_repo2": 268, "group_rank": 2 }, { "id": "Opus 4.8 (max effort)", "n_runs": 1, "model": "Opus 4.8", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-10", "color": "#E38365", "fixed": 15, "repo1_fixed": 6, "repo2_fixed": 9, "partial": 0, "claimed_only": 1, "extras": 3, "false_positive_fixes": 0, "wall_min": 108.8, "cost_usd": 58.11, "cost_kind": "list", "wall_note": "Ran alongside a leg on a different provider for part of its wall (cross-provider parallelism is allowed on this board; same-provider is not). Contention can inflate this figure, not shrink it. Scores, tokens and cost are unaffected.", "ceiling": true, "featured": false, "superseded": null, "caveat": "One run at this tier. The board's own variance evidence - three runs of one model at one setting scoring 16, 13 and 17 - is why a six-bug gap between two single runs is reported as what happened rather than as the size of the tier effect.", "note": "Opus 4.8's top tier, and the row exists to answer what the dial buys on a model the board already measured. It buys six bugs: the default-effort Opus 4.8 row is 9/105 (2/45 + 7/60), max is 15/105 (6/45 + 9/60), and the gain shows on both repos rather than on one. That is a real move and still leaves this model near the bottom of the frontier field - Opus 5 runs 21-27 across its tiers and Fable 5.1 runs 21-43 across its four. Both legs exited success with a report written, so nothing here is a harness failure. The price of the six bugs is the part worth reading twice: $58.11 at list for 15 fixes is $3.87 a fix - level with Fable 5 (max effort) for the worst cost-per-fix of any Anthropic row here, and about seven times what Opus 5.5 pays per fix at medium effort.", "fixed_bugs": [ 16, 20, 26, 30, 44, 45, 47, 49, 50, 60, 71, 76, 77, 92, 99 ], "turns": 469, "turns_repo1": 209, "turns_repo2": 260, "group_rank": 400 }, { "id": "GPT-6 Luna (xhigh effort)", "n_runs": 1, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "xhigh", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 14, "repo1_fixed": 4, "repo2_fixed": 10, "partial": 0, "claimed_only": 0, "extras": 12, "false_positive_fixes": 0, "wall_min": 57.0, "cost_usd": 0.39, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it does compare cleanly to the other GPT-6 rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs.", "note": "OpenAI's GPT-6 Luna at xhigh effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 14 of 105: 4 of 45 on repo 1 and 10 of 60 on repo 2, 12 genuine extras, 57.0 minutes, $0.39. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Luna ladder currently reads low 4, medium 4, high 9, xhigh 14, max 18.3 (n=3). Its own predecessor's ladder, GPT-5.6 Luna, publishes 31.3 (mean of 3) of 105 at max and 23 at this same rung, which is the generation-over-generation comparison at a fixed dial setting. *** WHY THE SWEEP EXISTS *** a single top-of-ladder number cannot tell a weak model from a mis-set dial. Running every rung of the same model on the same harness does: if the rungs separate, the dial reaches the model and the max figure is the model's ceiling. This family's dial has been steep - the GPT-6 Sol ladder spans roughly five times from low to max. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-luna, same pinned executable, same code-mode setting, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows quoted above. The GPT-6 Luna rungs DO compare cleanly to each other and to the GPT-6 Sol rungs - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million - twenty times cheaper than GPT-6 Sol on both sides of the meter. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 20, 30, 43, 45, 47, 49, 50, 62, 76, 77, 83, 84, 92, 99 ], "turns": 172, "turns_repo1": 73, "turns_repo2": 99, "group_rank": 301 }, { "id": "DeepSeek V4-Flash 0731", "n_runs": 1, "model": "DeepSeek V4-Flash", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-01", "color": "#B5A8E6", "fixed": 14, "repo1_fixed": 6, "repo2_fixed": 8, "partial": 1, "claimed_only": 2, "extras": 0, "false_positive_fixes": 0, "wall_min": 48.5, "cost_usd": 1.52, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "The Jul 31 re-post-trained revision.", "fixed_bugs": [ 13, 20, 25, 30, 39, 43, 50, 67, 71, 76, 83, 92, 97, 99 ], "turns": 205, "turns_repo1": 66, "turns_repo2": 139, "group_rank": 206 }, { "id": "GPT-6 Sol (medium effort)", "n_runs": 1, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 14, "repo1_fixed": 3, "repo2_fixed": 11, "partial": 0, "claimed_only": 1, "extras": 13, "false_positive_fixes": 0, "wall_min": 24.5, "cost_usd": 2.39, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; it does compare cleanly to the other GPT-6 Sol rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs, so the wall figure is not contended.", "note": "OpenAI's GPT-6 Sol at medium effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 14 of 105: 3 of 45 on repo 1 and 11 of 60 on repo 2, 13 genuine extras, 24.5 minutes, $2.39. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Sol ladder currently reads low 6, medium 14, high 20, xhigh 25, max 29.3 (n=3). For comparison the board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max and 43 at xhigh, and GPT-5.6 Sol at 43.5 (mean of 2) at max. This model's max rung replicated to three runs and landed below all of them, which is why the rungs beneath it are being measured at all: a sweep separates a weak model from a mis-set dial. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration, and this model's own max rung spans three runs. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-sol, same pinned executable, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows. The GPT-6 Sol rungs DO compare cleanly to each other - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for both legs, so these wall figures compare directly against other uncontended rows. (Corrected 2026-09-23: this row first published saying the run shared the laptop, which its receipts do not support - neither leg recorded a peer. The claim was a literal copied from a publish script, and the wall column is the evidence that the effort dial reaches the model at all, so it is worth getting right in both directions.)", "fixed_bugs": [ 13, 16, 30, 47, 49, 50, 77, 82, 83, 84, 98, 99, 100, 102 ], "turns": 103, "turns_repo1": 49, "turns_repo2": 54, "group_rank": 203 }, { "id": "Muse Spark 1.2 (default effort, OpenRouter)", "n_runs": 1, "model": "Muse Spark 1.2", "vendor": "Meta", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-06", "color": "#EDA3BD", "fixed": 14, "repo1_fixed": 3, "repo2_fixed": 11, "partial": 0, "claimed_only": 0, "extras": 3, "false_positive_fixes": 0, "wall_min": 65.2, "cost_usd": 19.52, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Aggregator run at an inert default effort, replaced by the Meta-API xhigh row, where the tier is verified and the ceiling is known.", "caveat": null, "note": null, "fixed_bugs": [ 20, 30, 43, 47, 60, 71, 76, 77, 83, 84, 92, 97, 99, 102 ], "turns": 424, "turns_repo1": 144, "turns_repo2": 280, "group_rank": 306 }, { "id": "GPT-5.6 Luna (high effort)", "n_runs": 1, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-07-31", "color": "#79C6AF", "fixed": 13, "repo1_fixed": 5, "repo2_fixed": 8, "partial": 2, "claimed_only": 0, "extras": 22, "false_positive_fixes": 0, "wall_min": 64.1, "cost_usd": 0.57, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 16, 17, 20, 30, 43, 47, 49, 62, 76, 77, 83, 84, 99 ], "turns": 174, "turns_repo1": 111, "turns_repo2": 63, "group_rank": 602 }, { "id": "GLM-5.3 Flash", "n_runs": 1, "model": "GLM-5.3 Flash", "vendor": "Z.ai", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-27", "color": "#55C9D8", "fixed": 13, "repo1_fixed": 6, "repo2_fixed": 7, "partial": 0, "claimed_only": 5, "extras": 4, "false_positive_fixes": 0, "wall_min": 57.1, "cost_usd": 0.79, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": "The named release of the Ox Alpha stealth slug, run on Z.ai's own host with provider fallbacks off. 13 against the stealth run's 16 is inside this board's same-setting spread. The bill is at the 50% launch price; list doubles after 9 Sep 2026. Its five claimed-only are bugs it named in its report and deliberately left alone.", "fixed_bugs": [ 20, 26, 27, 30, 39, 43, 50, 71, 76, 83, 97, 99, 102 ], "turns": 243, "turns_repo1": 91, "turns_repo2": 152, "group_rank": 106 }, { "id": "DeepSeek V4-Pro (high effort, DeepSeek API)", "n_runs": 1, "model": "DeepSeek V4-Pro", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / DeepSeek API", "effort": "high", "effort_status": "first_party", "date": "2026-09-10", "color": "#8773D7", "fixed": 13, "repo1_fixed": 4, "repo2_fixed": 9, "partial": 0, "claimed_only": 2, "extras": 1, "false_positive_fixes": 0, "wall_min": 26.6, "cost_usd": 1.08, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Two things to keep in view. (1) The effort label is `first_party`, not `verified`, for the same reason as the max row: a high-vs-max thinking sweep on Pro itself never separated - at n=5 the between-level ratio is 1.35x against a worst within-level spread of 2.27x, with the ranges overlapping heavily. The path-level facts do hold - the endpoint 400-rejects an invented tier and names its enum, and the served model read back as V4-Pro on every gate call. This bench pair is a second and coarser reading of the same question, and it does land in the expected direction (16 at max, 13 at high, with more extras and more money at max) - but it is n=1 per tier on 105 bugs, so it corroborates the dial rather than proving it. (2) This arm was run twice. The first attempt was VOIDED, not scored: a machine-wide tool hook on the bench host pointed at a moved script and denied every tool call the model made, so it diagnosed a broken environment, wrote no report, changed no file, and exited 0 after 160 seconds - a clean-looking row for a leg in which nothing happened. The numbers here come entirely from the re-run after that was fixed, and the runner now voids any leg that produces neither a report nor a single changed file.", "note": "The dial companion to the Pro max row: same model, same first-party endpoint, same hour, one tier down. It costs 3 strict fixes - 13/105 against 16 - and saves $0.81, $1.08 against $1.89, on 26.6 minutes against 37.2. It also moves where the score comes from: max is even across the two repos (8/45 and 8/60) while high collapses on repo 1 to 4/45 and edges ahead on repo 2 at 9/60. The unplanted-extras column tracks the money rather than the score - 4 genuine extras at max, 1 at high. high is DeepSeek's documented default; it is requested explicitly here rather than left implicit, because on this endpoint an omitted field measured as neither tier.", "fixed_bugs": [ 20, 25, 30, 43, 50, 60, 71, 76, 83, 84, 97, 98, 99 ], "turns": 288, "turns_repo1": 102, "turns_repo2": 186, "group_rank": 102 }, { "id": "Muse Spark 1.3 (medium effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "medium", "effort_status": "verified", "date": "2026-09-14", "color": "#E57EA2", "fixed": 13, "repo1_fixed": 5.3, "repo2_fixed": 7.7, "partial": 2.6, "claimed_only": 1, "extras": 17.3, "false_positive_fixes": 0, "wall_min": 58.3, "cost_usd": 9.24, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "A mean of three runs, not a single measurement; the three span 9 to 15. This row SUPERSEDES the single-run medium row (9/105) published 2026-09-13; three of that row's value and token claims are retracted above, and its dead-middle reading survives on a smaller margin. Every figure on this row is a MEAN of the three runs, carried to one decimal.", "note": "The second measured rung of Meta's first-party dial, published as the MEAN OF THREE INDEPENDENT RUNS. Runs scored 9, 15 and 15 of 105 (repo 1: 4, 4, 8 - repo 2: 5, 11, 7). It sits ONE NOTCH BELOW THIS ENDPOINT'S MEASURED DEFAULT: omitting reasoning.effort echoes high, so medium is a tier that has to be asked for and lands below the out-of-the-box run. *** WHY THIS RUNG WAS RUN THREE TIMES *** At n=1 low had measured 12 and the published medium row was 9, so low sat ABOVE medium. At n=3 the means are low 9.7 and medium 13.0 and that ordering reverses: the single medium run was the worst of medium's three. What that n=1 pair supported is therefore narrower than what was published from it. STILL STANDING: medium below high, so the dial does move score at the bottom while the 1-point high-to-xhigh step stays dead in the middle - the margin is 6 points, not the 10 it fired on. RETRACTED: that medium buys 'nearly the same money for less than half the result' (13 against 19 for 9.24 against 10.81), that medium is the worst buy on the board at 1 dollar 11 per fix (the mean is 71 cents), and that medium emits MORE reasoning than high (149,769 against 178,300 - that inversion was one verbose run). *** WHAT THE SPREAD ACTUALLY IS *** Three identical low runs scored 8, 5 and 2 on repo 1. That 6-point range is TWICE the 3-point gap from low's mean to high on the same repo, and on repo 1 low and medium tie at the mean (5.0 and 5.3). The within-tier spread is larger than the between-tier signal at the bottom of this dial. All six legs were clean: the Linux install returned 0, the shim logged zero errors, every leg exited 0, and no leg carries a handicap note. This is the model at this setting, not the harness. *** AND THE VARIANCE HAS A VISIBLE MECHANISM *** Same arm, same effort flag, repo 1: 221, 202 and 147 tool turns and 69,796, 45,411 and 40,238 reasoning tokens, scoring 8, 5 and 2 - monotone. The effort flag does not fix the amount of work done; the agent varied 1.5x in turns run to run and the score followed. Cost fell with it (3.77, 3.45, 2.83 dollars), so on that leg the cheap run is cheap BECAUSE it did less. SAID PRECISELY: that ordering is clean on repo 1's low legs and NOT on the other three - repo 2's low legs ran 167, 151 and 133 turns for 4, 8 and 2, and medium's most-active repo-1 leg scored lowest of its three. So the supported claim is the weaker one: the effort flag fixes a request field, not an amount of work, and the work actually performed varies by about a third inside one tier. It is not that turns predict score. *** READ THE REST OF THIS DIAL ACCORDINGLY *** high has since been run to n=3 as well and its mean is 18.7 (runs 19, 17, 20), so the single run published at that rung WAS the rung - replication confirmed it rather than overturning it. xhigh and max have since been run to n=3 as well (xhigh 20, 25, 16 - mean 20.3, range 9; max 33, 33, 35 - mean 33.7, range 2), so every rung on this dial is now a three-run mean and none of them is a single run. The three replicated rungs produced ranges of 9, 6 and 3 going up the dial. That is an OBSERVATION, not a finding: a range from three draws is a very noisy estimate of spread and three of them cannot establish a trend in variance. Do not read it as 'single runs are safer at the top of a dial'. With the bottom two rungs measured properly the dial is MONOTONE - 9.7, 13.0, 18.7, 20.3, 33.7 - with no dip and no dead middle. The dip was an artifact of n=1 at the two noisiest rungs. Medium costs about 1.6x low and returns about 1.34x the score, so the bottom of this dial is not a bargain: it is cheaper because it does less. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR: Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control. billing=LIST ESTIMATE, NOT A BILL: a subscription account with no per-run credits delta, costed at the same Meta list rates as the other first-party rows, which is what keeps them comparable. ctx 1M.", "fixed_bugs": [ 16, 20, 22, 25, 27, 30, 39, 43, 45, 47, 49, 50, 56, 60, 67, 71, 76, 77, 97, 98, 99, 100, 102 ], "bug_hits": { "16": 1, "20": 3, "22": 2, "25": 1, "27": 1, "30": 3, "39": 1, "43": 2, "45": 2, "47": 2, "49": 1, "50": 1, "56": 1, "60": 2, "67": 1, "71": 1, "76": 3, "77": 2, "97": 1, "98": 1, "99": 2, "100": 2, "102": 3 }, "coverage_runs": 3, "turns": 240, "turns_repo1": 88, "turns_repo2": 152, "group_rank": 203 }, { "id": "Grok 4.5 (high re-run - CLI has no max)", "n_runs": 1, "model": "Grok 4.5", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "high", "effort_status": "clamped", "date": "2026-08-01", "color": "#AAADB1", "fixed": 13, "repo1_fixed": 5, "repo2_fixed": 8, "partial": 3, "claimed_only": 1, "extras": 5, "false_positive_fixes": 0, "wall_min": 25.4, "cost_usd": 10.94, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Same setting as the other Grok 4.5 runs.", "caveat": "Commissioned as a max run; the CLI silently clamps unknown tiers, so this is a second high run - published as a correction.", "note": null, "fixed_bugs": [ 20, 29, 30, 33, 34, 47, 49, 50, 62, 76, 84, 99, 100 ], "turns": null, "turns_repo1": null, "turns_repo2": null, "group_rank": 202 }, { "id": "Haiku 5.5 (medium effort)", "n_runs": 1, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "medium", "effort_status": "first_party", "date": "2026-10-08", "color": "#EBA691", "fixed": 12, "repo1_fixed": 5, "repo2_fixed": 7, "partial": 0, "claimed_only": 2, "extras": 4, "false_positive_fixes": 0, "wall_min": 15.5, "cost_usd": 2.66, "cost_kind": "list", "wall_note": "Four legs ran at the same time on one account (both repos of this run and of a parallel run); contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": null, "caveat": "One run at medium effort, run with four legs at once on one account (both repos of this run and of the other arm). Single runs on this board move by more than 2 points against a three-run mean about half the time. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at medium effort through Claude Code on the first-party API, released 2026-10-07. ONE RUN: 12 of 105 - 5 of 45 on repo 1 and 7 of 60 on repo 2 - with 0 partial fix(es), 2 claimed-only and 4 genuine unplanted extras; 15.5 minutes, $2.66. The same model's xhigh row is a mean of three runs; this is a single run, and on this board a first run has moved by more than 2 points against its own three-run mean about half the time. *** EFFORT: MEDIUM - THE MODEL'S API DEFAULT - *** a first-party Claude Code tier. Not captured on the wire for this run: the xhigh and max captures on the same CLI build showed the flag passed through verbatim for this model, and medium goes through the same path. Accepted, not magnitude-probed. Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** FOUR LEGS AT ONCE *** both repo legs of this run AND both legs of the parallel medium/low arm ran at the same time on one account, so each leg's wall includes the others' contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and each leg's priced responses were checked to be exactly the ones in its own stream, with none shared among all four legs. *** COST, PRICED PER REQUEST *** 106 of 118 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates; 97.6% of prompt tokens were cache reads. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 16, 25, 30, 43, 45, 49, 50, 71, 76, 77, 84, 99 ], "turns": 194, "turns_repo1": 65, "turns_repo2": 129, "group_rank": 703 }, { "id": "Haiku 5.5 (low effort)", "n_runs": 1, "model": "Haiku 5.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "low", "effort_status": "first_party", "date": "2026-10-08", "color": "#EBA691", "fixed": 11, "repo1_fixed": 4, "repo2_fixed": 7, "partial": 0, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 8.7, "cost_usd": 1.61, "cost_kind": "list", "wall_note": "Four legs ran at the same time on one account (both repos of this run and of a parallel run); contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": null, "caveat": "One run at low effort, run with four legs at once on one account (both repos of this run and of the other arm). Single runs on this board move by more than 2 points against a three-run mean about half the time. Cost is a list estimate priced per request at Haiku 5.5's prompt-length tiers.", "note": "Anthropic's Claude Haiku 5.5 at low effort through Claude Code on the first-party API, released 2026-10-07. ONE RUN: 11 of 105 - 4 of 45 on repo 1 and 7 of 60 on repo 2 - with 0 partial fix(es), 0 claimed-only and 2 genuine unplanted extras; 8.7 minutes, $1.61. The same model's xhigh row is a mean of three runs; this is a single run, and on this board a first run has moved by more than 2 points against its own three-run mean about half the time. *** EFFORT: LOW *** a first-party Claude Code tier. Not captured on the wire for this run: the xhigh and max captures on the same CLI build showed the flag passed through verbatim for this model, and low goes through the same path. Accepted, not magnitude-probed. Claude Code's one per-session title call runs at high, which the effort flag does not reach. *** FOUR LEGS AT ONCE *** both repo legs of this run AND both legs of the parallel medium/low arm ran at the same time on one account, so each leg's wall includes the others' contention. Scores and costs did not mix: each leg had its own run dir, stream, judge pass and session record, and each leg's priced responses were checked to be exactly the ones in its own stream, with none shared among all four legs. *** COST, PRICED PER REQUEST *** 84 of 96 requests were over Haiku 5.5's 100,000-token prompt tier and priced at its upper rates; 96.8% of prompt tokens were cache reads. A list estimate, not a bill. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 25, 30, 43, 45, 49, 71, 76, 84, 97, 98, 99 ], "turns": 127, "turns_repo1": 53, "turns_repo2": 74, "group_rank": 704 }, { "id": "Grok 4.7 (low)", "n_runs": 1, "model": "Grok 4.7", "vendor": "xAI", "local": false, "harness": "Grok Build CLI (ACP)", "effort": "low", "effort_status": "verified", "date": "2026-09-21", "color": "#414852", "fixed": 11, "repo1_fixed": 5, "repo2_fixed": 6, "partial": 0, "claimed_only": 0, "extras": 12, "false_positive_fixes": 0, "wall_min": 15.1, "cost_usd": 5.41, "cost_kind": "floor", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-21 by the n=3 row at the same configuration (15.7 of 105). The three runs scored 11, 21 and 15, so this single draw at 11 sat at the edge of a 10-point spread it could not show. Nothing it claimed is retracted; it was published as an n=1 row and read as one. Kept rather than deleted because it is the receipt for what the board showed for part of 2026-09-21.", "caveat": "ONE RUN, not a mean. The same model at its top rung spanned 25 to 31 of 105 across three runs, so read this number with several points of slack in either direction. Cost is a reconstructed FLOOR rather than a bill, and it uses a per-session arithmetic the older grok rows on this board do not yet use; the note gives both figures.", "note": "Grok 4.7 at low through xAI's own CLI - the LOW rung, the floor of the effort range - and the rung where the 4.6 dial had the most room, scoring 15 of 105 against its own 28.7 at the ceiling. 11 of 105: 5 of 45 on repo 1 and 6 of 60 on repo 2, 12 genuine extras, 15.1 minutes, a reconstructed floor of $5.41. ONE RUN. *** AGAINST THE PREVIOUS GENERATION AT THE SAME RUNG *** Grok 4.6 (low) scored 15.0 of 105 (6 on repo 1, 9 on repo 2), also a single run, for a difference of -4. Both sides are n=1, so this is one draw against one draw. The scale of the doubt is not a guess: three runs of the 4.6 configuration one rung up spanned 27 to 30 of 105, and three runs of this model at that rung spanned 25 to 31 - so a gap of a few points between two single runs is inside what the same configuration produces run to run. *** WHERE IT SITS ON ITS OWN DIAL *** This model at xhigh is 28.8 of 105, published as a mean of four. This rung is 11, a single run. The 4.6 dial for comparison ran low 15, medium 23, high 23, xhigh 28.7 - FLAT between medium and high, which is the most interesting thing on it, and the low rung is where that dial had real room. Whether either shape repeats here is a question the rungs answer together: read this row beside its siblings rather than on its own. *** THE TIER WAS CHECKED, NOT ASSUMED, AND FOR THE LOWER RUNGS THAT MATTERS MORE *** A zero-token ACP readback before launch returned reasoning_effort low for a requested low. The grok CLI silently clamps an unknown effort string AND its default is `high`, so a rung that failed to apply would look identical to one that worked in every artefact except this readback - the run would simply be a default-effort run wearing another label. The enum grok-4.7 advertises is low, medium, high, xhigh. The served model is named by the session itself (currentModelId grok-4.7) rather than inferred from the -m flag, which has been inert on this bridge since it was probed on 2026-09-04. *** COST IS A FLOOR *** The grok CLI reports context occupancy rather than cumulative usage, so every grok row here is reconstructed from the CLI's own session logs at xAI list rates, the same 2/0.5/6 per MTok basis. This row uses the per-session arithmetic corrected on 2026-09-21; on the older run-level arithmetic the other grok rows still use it would read $5.41. It is a floor twice over: xAI bills a request whose prompt reaches 200K at DOUBLE those rates. *** RECEIPTS *** Both legs exited 0. Legs SEQUENTIAL, one repo at a time, uncontended - nothing else ran alongside. The agent spawned 1 session(s) on repo 1 and 1 on repo 2 (43 and 46 inference requests); that count varies run to run on identical arm definitions for this model, and it tracks cost far more closely than score. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 20, 30, 37, 39, 45, 60, 76, 83, 84, 99, 100 ], "turns": 253, "turns_repo1": 101, "turns_repo2": 152, "group_rank": 4 }, { "id": "Qwen3.8-27B 8-bit (OpenRouter/Parasail) - mean of 3", "n_runs": 3, "model": "Qwen3.8-27B 8-bit", "vendor": "Alibaba", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-16", "color": "#F0C372", "fixed": 10.7, "repo1_fixed": 3.7, "repo2_fixed": 7, "partial": 0.3, "claimed_only": 0, "extras": 1.7, "false_positive_fixes": 0, "wall_min": 56.6, "cost_usd": 2.97, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Every figure on this row is a MEAN of three runs; the single run this replaces (8/105, published 2026-09-15) sat at the low end of a 8-12 spread rather than near the mean, so read the 10.7 as the better estimate of the three, not as confirmation the 8 was typical. This row SUPERSEDES the single-run row published 2026-09-15. effort_status stays inert_default for the same reason as the n=1 row: OpenRouter does not implement output_config, so no tier was requested on any of the three runs and none is claimed. The [1m] context declaration overstates Parasail's real 262K ceiling on all three legs, same as the n=1 row - it never reaches the wire and only sets the CLI's local compaction arithmetic; see the arm definition for the full tradeoff. NOT FEATURED: mean rows are no longer featured by default on this board (reversed 2026-09-16) - featuring is an opt-in now, same as any single-run row, and this one was not opted in.", "note": "REPLICATION MOVES THIS ROW UP, NOT SIDEWAYS. Runs scored 8, 12 and 12 of 105 (repo 1: 2, 6, 3 - repo 2: 6, 6, 9). Mean 10.7, range 4. The single run published 2026-09-15 (8/105) was the LOW end of the spread the other two legs reveal, not a representative draw - the mean sits 2.7 points above it. *** ZERO OVER-CLAIMS ACROSS ALL SIX LEGS *** every one of the three runs reported claimed_only = 0 on both repos: nothing it said it fixed went unconfirmed by the blind judge, the same discipline the n=1 note flagged and replication holds it. One partial credit appears exactly once, on the original n=1 abhb leg; every other hit across the six legs is a full match. *** GENUINE UNPLANTED FIXES ON FIVE OF SIX LEGS *** extras (real defects nobody planted) came in at 1, 1 on the n=1 run, 0, 2 on r2, and 0, 1 on r3 (repo 1 / repo 2 each) - mean 1.7 across the six legs, never zero on more than one leg at a time. This model reliably finds things outside the answer key even when it is missing most of what is in it. *** REGIME: SAME on both repos, VERIFIED not assumed *** this route (OpenRouter -> Parasail, via a local shim with the CLI given a declared [1m] context) reports real context/compaction accounting, unlike the gpt-oss OpenRouter arms on this board (CANNOT TELL there) - the harness check ran and passed rather than being unreadable. Per-leg receipts (repo 1 / repo 2): wall 37.4/21.6, 41.9/30.2 and 13.1/25.7 minutes; cost 1.05/1.06, 2.63/2.10 and 0.70/1.38 dollars, all six REAL OpenRouter credits-delta bills, not estimates. Combined mean cost across all six legs: $2.97. BILLING: real OpenRouter bill (credits-delta), matching the n=1 row - not a list-rate estimate.", "fixed_bugs": [ 16, 20, 25, 30, 39, 41, 43, 45, 50, 60, 62, 71, 76, 83, 92, 97, 98, 99 ], "bug_hits": { "16": 1, "20": 2, "25": 1, "30": 2, "39": 1, "41": 1, "43": 2, "45": 1, "50": 2, "60": 1, "62": 1, "71": 3, "76": 3, "83": 3, "92": 1, "97": 1, "98": 3, "99": 3 }, "coverage_runs": 3, "turns": 329, "turns_repo1": 153, "turns_repo2": 177, "group_rank": 206 }, { "id": "DeepSeek V4-Pro", "n_runs": 1, "model": "DeepSeek V4-Pro", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-08-01", "color": "#8773D7", "fixed": 10, "repo1_fixed": 5, "repo2_fixed": 5, "partial": 0, "claimed_only": 1, "extras": 1, "false_positive_fixes": 0, "wall_min": 27.6, "cost_usd": 5.54, "cost_kind": "bill", "wall_note": null, "ceiling": true, "featured": false, "superseded": "Aggregator run at an inert default effort. The Sep 10 rows run the same model version on DeepSeek's own API with the tier asserted and the served model id read back - V4-Pro at max (16/105) and at high (13/105) - and they replace this one.", "caveat": null, "note": null, "fixed_bugs": [ 13, 16, 20, 39, 43, 76, 77, 83, 84, 99 ], "turns": 184, "turns_repo1": 69, "turns_repo2": 115, "group_rank": 106 }, { "id": "Muse Spark 1.3 (low effort, Muse Code, Meta API) - mean of 3", "n_runs": 3, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "low", "effort_status": "verified", "date": "2026-09-14", "color": "#E57EA2", "fixed": 9.7, "repo1_fixed": 5, "repo2_fixed": 4.7, "partial": 1.3, "claimed_only": 1.6, "extras": 6.7, "false_positive_fixes": 0, "wall_min": 36.4, "cost_usd": 5.61, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "A mean of three runs, not a single measurement - and the three disagree by 9 points of 105, which is larger than most adjacent-rung gaps on this board. Quote the mean only with the spread. Every figure on this row is a MEAN of the three runs, carried to one decimal - 9.7 of 105. A column that counts bugs showing 9.7 is the point: no run scored 9.7, and no single run is the rung.", "note": "The BOTTOM measured rung of Meta's first-party dial, published as the MEAN OF THREE INDEPENDENT RUNS rather than as one run. Runs scored 12, 13 and 4 of 105 (repo 1: 8, 5, 2 - repo 2: 4, 8, 2). *** WHY THIS RUNG WAS RUN THREE TIMES *** At n=1 low had measured 12 and the published medium row was 9, so low sat ABOVE medium. At n=3 the means are low 9.7 and medium 13.0 and that ordering reverses: the single medium run was the worst of medium's three. What that n=1 pair supported is therefore narrower than what was published from it. STILL STANDING: medium below high, so the dial does move score at the bottom while the 1-point high-to-xhigh step stays dead in the middle - the margin is 6 points, not the 10 it fired on. RETRACTED: that medium buys 'nearly the same money for less than half the result' (13 against 19 for 9.24 against 10.81), that medium is the worst buy on the board at 1 dollar 11 per fix (the mean is 71 cents), and that medium emits MORE reasoning than high (149,769 against 178,300 - that inversion was one verbose run). *** WHAT THE SPREAD ACTUALLY IS *** Three identical low runs scored 8, 5 and 2 on repo 1. That 6-point range is TWICE the 3-point gap from low's mean to high on the same repo, and on repo 1 low and medium tie at the mean (5.0 and 5.3). The within-tier spread is larger than the between-tier signal at the bottom of this dial. All six legs were clean: the Linux install returned 0, the shim logged zero errors, every leg exited 0, and no leg carries a handicap note. This is the model at this setting, not the harness. *** AND THE VARIANCE HAS A VISIBLE MECHANISM *** Same arm, same effort flag, repo 1: 221, 202 and 147 tool turns and 69,796, 45,411 and 40,238 reasoning tokens, scoring 8, 5 and 2 - monotone. The effort flag does not fix the amount of work done; the agent varied 1.5x in turns run to run and the score followed. Cost fell with it (3.77, 3.45, 2.83 dollars), so on that leg the cheap run is cheap BECAUSE it did less. SAID PRECISELY: that ordering is clean on repo 1's low legs and NOT on the other three - repo 2's low legs ran 167, 151 and 133 turns for 4, 8 and 2, and medium's most-active repo-1 leg scored lowest of its three. So the supported claim is the weaker one: the effort flag fixes a request field, not an amount of work, and the work actually performed varies by about a third inside one tier. It is not that turns predict score. *** READ THE REST OF THIS DIAL ACCORDINGLY *** high has since been run to n=3 as well and its mean is 18.7 (runs 19, 17, 20), so the single run published at that rung WAS the rung - replication confirmed it rather than overturning it. xhigh and max have since been run to n=3 as well (xhigh 20, 25, 16 - mean 20.3, range 9; max 33, 33, 35 - mean 33.7, range 2), so every rung on this dial is now a three-run mean and none of them is a single run. The three replicated rungs produced ranges of 9, 6 and 3 going up the dial. That is an OBSERVATION, not a finding: a range from three draws is a very noisy estimate of spread and three of them cannot establish a trend in variance. Do not read it as 'single runs are safer at the top of a dial'. Each leg is a full independent run of both repos; the mean is over three, and the per-repo runs are stated so the spread stays visible - it is the finding, not a caveat on it. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR: Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control. billing=LIST ESTIMATE, NOT A BILL: a subscription account with no per-run credits delta, costed at the same Meta list rates as the other first-party rows, which is what keeps them comparable. ctx 1M.", "fixed_bugs": [ 16, 20, 22, 25, 30, 39, 43, 44, 45, 60, 62, 76, 77, 83, 84, 95, 97, 98, 99, 100, 102 ], "bug_hits": { "16": 2, "20": 3, "22": 1, "25": 1, "30": 3, "39": 1, "43": 1, "44": 1, "45": 2, "60": 1, "62": 1, "76": 1, "77": 2, "83": 1, "84": 2, "95": 1, "97": 1, "98": 1, "99": 1, "100": 1, "102": 1 }, "coverage_runs": 3, "turns": 175, "turns_repo1": 93, "turns_repo2": 82, "group_rank": 204 }, { "id": "GPT-6 Luna (high effort)", "n_runs": 1, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "high", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 9, "repo1_fixed": 2, "repo2_fixed": 7, "partial": 2, "claimed_only": 0, "extras": 9, "false_positive_fixes": 0, "wall_min": 24.4, "cost_usd": 0.13, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it does compare cleanly to the other GPT-6 rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs.", "note": "OpenAI's GPT-6 Luna at high effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 9 of 105: 2 of 45 on repo 1 and 7 of 60 on repo 2, 9 genuine extras, 24.4 minutes, $0.13. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Luna ladder currently reads low 4, medium 4, high 9, xhigh 14, max 18.3 (n=3). Its own predecessor's ladder, GPT-5.6 Luna, publishes 31.3 (mean of 3) of 105 at max and 13 at this same rung, which is the generation-over-generation comparison at a fixed dial setting. *** WHY THE SWEEP EXISTS *** a single top-of-ladder number cannot tell a weak model from a mis-set dial. Running every rung of the same model on the same harness does: if the rungs separate, the dial reaches the model and the max figure is the model's ceiling. This family's dial has been steep - the GPT-6 Sol ladder spans roughly five times from low to max. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-luna, same pinned executable, same code-mode setting, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows quoted above. The GPT-6 Luna rungs DO compare cleanly to each other and to the GPT-6 Sol rungs - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million - twenty times cheaper than GPT-6 Sol on both sides of the meter. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 30, 45, 49, 76, 77, 83, 84, 99, 100 ], "turns": 72, "turns_repo1": 29, "turns_repo2": 43, "group_rank": 302 }, { "id": "GPT-5.6 Luna (medium effort)", "n_runs": 1, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-08-27", "color": "#79C6AF", "fixed": 9, "repo1_fixed": 5, "repo2_fixed": 4, "partial": 0, "claimed_only": 0, "extras": 5, "false_positive_fixes": 0, "wall_min": 15.4, "cost_usd": 0.33, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 20, 22, 30, 37, 43, 76, 77, 83, 99 ], "turns": 97, "turns_repo1": 58, "turns_repo2": 39, "group_rank": 603 }, { "id": "GLM-5.3 Flash (low effort, Z.ai API)", "n_runs": 1, "model": "GLM-5.3 Flash", "vendor": "Z.ai", "local": false, "harness": "Claude Code / Z.ai API", "effort": "low", "effort_status": "verified", "date": "2026-09-13", "color": "#55C9D8", "fixed": 9, "repo1_fixed": 7, "repo2_fixed": 2, "partial": 0, "claimed_only": 2, "extras": 4, "false_positive_fixes": 0, "wall_min": 29.0, "cost_usd": 0.42, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "One run at each tier, so the 10-point gap is n=1 against n=1 - and this board declined to credit a 6-point gap on the max row for exactly that reason, which is worth explaining rather than glossing. The difference is what else moved. That 6-point gap spanned a change of ROUTE and TIER together with a control arguing against the tier; this one holds route, harness, model, prompt and repos fixed and moves one field by one notch, and it is larger. A repeat at either tier would still turn a strong observation into a finding. WHAT IS NOT CLAIMED: anything about high, which is the rung between these two and is running as this row publishes. Nor that 9 is this model's floor - 2 of 60 on repo 2 is close to the bottom of this board, and a second run at low could land either side of it. billing=LIST ESTIMATE, NOT A BILL: Z.ai publishes no balance or usage endpoint on this plan key, so no credits delta exists to read; costed at the same published Flash rates the OpenRouter Flash row was billed at, which keeps all three Flash rows comparable - and those rates carry a launch discount that can lapse. This endpoint does not report reasoning tokens separately, so output volume is the only work proxy available for the pair. ctx 1M.", "note": "The BOTTOM of Z.ai's three-tier dial for this family, run first-party on the same endpoint as the published max row - low, high and max are the entire accept-list here, medium does not exist and cannot be run, and max is also Z.ai's documented default, so low is a tier that has to be asked for and sits BELOW the out-of-the-box run. *** THE DIAL BINDS ON SCORE, AND THIS IS THE FIRST SINGLE-VARIABLE TIER PAIR THIS BOARD HAS RUN ON Z.AI *** 9 of 105 here against 19 at max: same endpoint, same harness, same model, one notch of one field between them. THE TIER IS WORTH TEN POINTS. It shows on the work volume too - 36,777 output tokens against the max run's 115,653, a 3.1x gap - and on the clock, 29.0 minutes against 58.4. *** AND IT ANSWERS A CAUTION THE max ROW PUBLISHED, AGAINST THAT ROW'S READING *** The max row scored 19 against the OpenRouter Flash row's 13 and said in as many words: do not read the jump as an effort result, read 19 as where Flash lands first-party rather than as what max bought. Its reason was the full-size GLM-5.3 control, which scored 19 on BOTH routes. That caution was the honest state of the evidence and the single-variable leg did not exist. It exists now and the dial moves this model by ten points, so the OpenRouter row's 13 sits INSIDE the range the tier alone covers. THE CONTROL IS NOT RETRACTED: the full-size model has no tier pair on this board, only a route pair, so the zero it moved is a statement about its ROUTE and says nothing about its dial, which nobody has run. *** WHERE IT BREAKS IS REPO 2, ALMOST ENTIRELY *** Repo 1 gives up 3 points, 7 of 45 here against 10 at max. Repo 2 nearly collapses: 2 of 60 against 9. The larger and more tangled codebase is where the missing thinking is missed, which is the same place every cheap row on this board loses its points, and it is worth more than the combined number. *** THE PRICE IS HONEST, WHICH IS THE OPPOSITE OF THE OTHER DIAL PUBLISHED TODAY *** Cost per strict fix: 4.6 cents at low, 5.0 cents at max. You pay 2.2x and you get 2.1x, so the bottom rung is not a value play - it is proportionally less of everything. Compare Muse Spark 1.3 on Meta's endpoint, published hours earlier, where xhigh costs 44% more than high and buys a single point. Two dials, two entirely different value shapes, same method. Honesty profile: 7 of 45 on repo 1 with no partials and no claimed-only, 2 of 60 on repo 2 with no partials and TWO claimed-only, and 4 genuine unplanted extras. Stated flat rather than framed: that is up from one claimed-only on the max row and down from five on the OpenRouter Flash row, so it is inside this model's own range and is not evidence that the bottom rung is less honest.", "fixed_bugs": [ 16, 20, 25, 27, 29, 39, 43, 71, 84 ], "turns": 155, "turns_repo1": 60, "turns_repo2": 95, "group_rank": 104 }, { "id": "Muse Spark 1.3 (medium effort, Muse Code, Meta API)", "n_runs": 1, "model": "Muse Spark 1.3", "vendor": "Meta", "local": false, "harness": "Muse Code / Meta API", "effort": "medium", "effort_status": "verified", "date": "2026-09-13", "color": "#E57EA2", "fixed": 9, "repo1_fixed": 4, "repo2_fixed": 5, "partial": 4, "claimed_only": 1, "extras": 8, "false_positive_fixes": 0, "wall_min": 61.7, "cost_usd": 10.02, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-14 by the n=3 MEAN row at the same rung (13/105). This row is ONE run of three, and it is the lowest of the three: medium scored 9, 15 and 15 on identical runs, and low scored 12, 13 and 4. THREE CLAIMS IN THE NOTE BELOW ARE RETRACTED. (1) 'Nearly the same money for less than half the result' - at the mean it is 13 against high's 19 for 9.24 dollars against 10.81. Cheaper and worse, but not half. (2) '1 dollar 11 per fix, the worst buy by a factor of two' - the mean is 71 cents per fix, mid-pack. (3) 'Reasoning volume does not explain it, medium emitted MORE than high' - the medium mean is 149,769 reasoning tokens against high's 178,300, so that inversion was one verbose run, not a property of the tier. TWO CLAIMS SURVIVE. On repo 2 medium still costs MORE in absolute dollars than high at the mean, 5.05 against 3.98, for 8 of 60 against 11 of 60. And the pre-registered branch still stands: 13 is still well below 19, so the dial does move score at the bottom while high-to-xhigh stays dead in the middle - it fired on a 10-point margin and the real one is 6. One more artifact of n=1: this run paired against a single low run of 12 put low ABOVE medium, and at n=3 the means are low 9.7 and medium 13.0, so that inversion was noise. Single runs at the bottom of this dial disagree by up to 9 points of 105. Kept rather than deleted because it is the receipt for what this board published on 2026-09-13, and because a retraction is only legible next to the claim it retracts.", "caveat": "One run at each of the four tiers, so every step on this dial is n=1 against n=1. Four rungs ordering correctly is more than any single pair would give, but a repeat at any rung could move it, and the 1-point high-to-xhigh step is inside the noise this board would not credit in either direction. THE 10-POINT GAP TO high IS THE LARGEST RELATIVE STEP ON THIS DIAL, NOT THE LARGEST ABSOLUTE ONE - said precisely because the difference matters. Score more than doubles across this one notch, 2.11 times, against 1.65 times for xhigh to max; in raw points xhigh to max is the bigger move, 13 against 10. What makes this the step worth trusting is not its size but that it REPRODUCES ON BOTH REPOS INDEPENDENTLY - 8 to 4 on repo 1, 11 to 5 on repo 2, each one roughly halving. A gap that appears twice on two unrelated codebases is not a coin flip. WALL CLOCK ON THIS ROW CARRIES THROTTLING THE OTHER RUNGS DID NOT SEE. The shim absorbed 19 upstream 429s across the two legs against 4 apiece on the high, xhigh and max arms. Retries are transparent to the model and cost wall clock rather than correctness, but 61.7 minutes here is not cleanly comparable with 64.5 at high for that reason, and the comparison this row actually rests on is cost and score, not wall. THE EFFORT LABEL DESCRIBES THE MAIN LOOP, NOT EVERY CALL THIS ROW IS BILLED FOR. Muse Code runs a reminder-observer side agent at fixed efforts the flag does not control, so a share of these calls carried a tier nobody asked for - the same caveat every Muse Code row on this board carries, and the reason the dial is read on the main loop that does the work. billing=LIST ESTIMATE, NOT A BILL: this runs on a subscription account with no per-run credits delta to read; costed at the same Meta list rates as the other three first-party rows, which is what keeps the four comparable. ctx 1M.", "note": "The BOTTOM measured rung of Meta's first-party dial, and the row that makes this the worst-value dial on the board. It sits ONE NOTCH BELOW THIS ENDPOINT'S MEASURED DEFAULT: omitting reasoning.effort echoes high, so medium is a tier that has to be asked for and lands below the out-of-the-box run. *** THE DIAL IS MONOTONIC ON SCORE ACROSS FOUR RUNGS *** 9 at medium, 19 at high, 20 at xhigh, 33 at max. Four first-party rows, one serving path, model and prompt and repos held fixed. No other model on this board has four bench-measured tiers. *** AND THE VALUE CURVE IS THE WORST THIS BOARD HAS SEEN *** Cost per strict fix runs 1 dollar 11 at medium, 57 cents at high, 78 cents at xhigh, 60 cents at max. THE BOTTOM RUNG IS THE WORST BUY BY A FACTOR OF TWO. medium is not a cheap tier that buys less - it costs 10.02 dollars against high's 10.81, which is 93 cents on the dollar, and returns 9 points against 19. Nearly the same money for less than half the result. *** ON REPO 2 IT IS WORSE THAN NEARLY-THE-SAME: IT COSTS MORE IN ABSOLUTE DOLLARS *** 5.22 against high's 3.98, a 31 percent premium, for 5 of 60 against 11 of 60. More money, less than half the score, one notch of one field between the two runs. *** REASONING VOLUME DOES NOT EXPLAIN IT, WHICH IS THE FINDING FOR ANYONE PROBING A DIAL *** Total reasoning tokens are 187,162 here against 178,300 at high - medium is HIGHER, not lower. Per leg the direction is not even consistent: repo 1 down 5 percent, repo 2 up 24 percent, while the score halves on both. A token-volume probe pointed at this pair would have reported no degradation, or an improvement. Whatever the tier changed, it is not how much the model thought. *** IT RESOLVES A PREDICTION REGISTERED IN WRITING BEFORE THE RUN *** The magnitude probe was falsified twice on the top half of this dial. Its one surviving claim was that the bottom three tiers separate cleanly in TOKEN terms - medium's ceiling of 6,237 reasoning tokens per call below high's floor of 7,824 - and that claim had never been tested against score. The arm was launched with both outcomes written down: near 19 would mean a disjoint token band predicts nothing, well below 19 would mean the dial moves score at the bottom and high/xhigh is a dead zone in the middle. 9 is well below 19. The second branch stands. Honesty profile: 4 of 45 on repo 1 and 5 of 60 on repo 2, TWO partials on each, ONE claimed-only in total, and 8 genuine unplanted extras against high's 10 - the same band. The claim record does not degrade with the tier.", "fixed_bugs": [ 20, 30, 43, 45, 76, 77, 97, 99, 102 ], "turns": 228, "turns_repo1": 79, "turns_repo2": 149, "group_rank": 203 }, { "id": "Sonnet 5", "n_runs": 1, "model": "Sonnet 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-07-26", "color": "#E99B82", "fixed": 9, "repo1_fixed": 1, "repo2_fixed": 8, "partial": 0, "claimed_only": 0, "extras": 4, "false_positive_fixes": 0, "wall_min": 32.8, "cost_usd": 11.38, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 22, 66, 70, 71, 76, 77, 83, 97, 99 ], "turns": 258, "turns_repo1": 143, "turns_repo2": 115, "group_rank": 602 }, { "id": "Sonnet 5 (max effort)", "n_runs": 1, "model": "Sonnet 5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "max", "effort_status": "first_party", "date": "2026-09-10", "color": "#E99B82", "fixed": 9, "repo1_fixed": 3, "repo2_fixed": 6, "partial": 1, "claimed_only": 0, "extras": 3, "false_positive_fixes": 0, "wall_min": 61.3, "cost_usd": 17.96, "cost_kind": "list", "wall_note": "Ran alongside a leg on a different provider for part of its wall (cross-provider parallelism is allowed on this board; same-provider is not). Contention can inflate this figure, not shrink it. Scores, tokens and cost are unaffected.", "ceiling": true, "featured": false, "superseded": null, "caveat": "A null from one run per setting is not proof the tier is inert - it is the absence of evidence that it is active, on a benchmark where this model finds few bugs at either setting, which is exactly the regime where a real effect would be hardest to see.", "note": "The clearest null on the board's Anthropic dial: Sonnet 5 at max scores 9/105, exactly what the default-effort Sonnet 5 row scored. The totals match and the split does not - default was 1/45 + 8/60, max is 3/45 + 6/60 - so the tier moved two bugs onto repo 1 and two off repo 2 and finished where it started. Two runs cannot separate that from ordinary run-to-run noise, and the honest reading is that asking Sonnet 5 for maximum effort bought nothing measurable on this benchmark. Both legs exited success with a report written. Cost rose about 58 per cent for it: $17.96 at list against $11.38 for the default row.", "fixed_bugs": [ 33, 34, 43, 76, 83, 92, 97, 99, 100 ], "turns": 411, "turns_repo1": 207, "turns_repo2": 204, "group_rank": 600 }, { "id": "Opus 4.8", "n_runs": 1, "model": "Opus 4.8", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "high", "effort_status": "first_party", "date": "2026-07-26", "color": "#E38365", "fixed": 9, "repo1_fixed": 2, "repo2_fixed": 7, "partial": 1, "claimed_only": 0, "extras": 1, "false_positive_fixes": 0, "wall_min": 34.8, "cost_usd": 21.3, "cost_kind": "list", "wall_note": null, "ceiling": true, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 20, 30, 50, 60, 71, 76, 83, 92, 99 ], "turns": 165, "turns_repo1": 97, "turns_repo2": 68, "group_rank": 402 }, { "id": "DeepSeek V4-Flash", "n_runs": 1, "model": "DeepSeek V4-Flash", "vendor": "DeepSeek", "local": false, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-07-31", "color": "#B5A8E6", "fixed": 8, "repo1_fixed": 4, "repo2_fixed": 4, "partial": 1, "claimed_only": 0, "extras": 4, "false_positive_fixes": 0, "wall_min": 24.9, "cost_usd": 0.61, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": "This slug resolved to the April snapshot; the Jul 31 revision is the row above.", "caveat": null, "note": null, "fixed_bugs": [ 20, 30, 39, 43, 47, 71, 84, 99 ], "turns": 190, "turns_repo1": 63, "turns_repo2": 127, "group_rank": 206 }, { "id": "Qwen3.8-27B 8-bit (OpenRouter/Parasail)", "n_runs": 1, "model": "Qwen3.8-27B 8-bit", "vendor": "Alibaba", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-15", "color": "#F0C372", "fixed": 8, "repo1_fixed": 2, "repo2_fixed": 6, "partial": 1, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 59.0, "cost_usd": 2.11, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-16 by the n=3 MEAN row (10.7/105, range 4 - the mean moves UP from the single run, which was the low end of the spread, not a typical draw). Nothing here is retracted: the 8 happened and this is its receipt. Kept rather than deleted because a supersession is only legible next to the row it supersedes.", "caveat": "ONE RUN, against a control that has three. The first-party row is a mean of 15, 13 and 17; this is a single 8 with no measured spread of its own, so the gap between them is known far less precisely than two numbers side by side suggest. billing=REAL BILL, and the control's is NOT: this figure is a credits delta read from OpenRouter before and after each leg, while the first-party row is a token-count estimate at list rates, because that endpoint publishes no usage API. A bill and an estimate are not the same kind of number and the 2.11-against-5.55 comparison should be read with that in mind. Token counts are NOT quoted for this row at all: for an arm behind a shim the local CLI invents them from an Anthropic price table, so they describe nothing real. EFFORT IS NOT ASSERTED. The first-party sibling runs at a MEASURED xhigh - that endpoint implements the effort field and rejects invalid values. OpenRouter does not implement it, so no tier was requested and none is claimed; the model reasons on its own regardless. THE CONTEXT DECLARATION OVERSTATES THE HOST, DELIBERATELY AND VISIBLY: the arm declares a 1M window to the CLI because through a shim that is the ONLY override the CLI accepts - every numeric alternative is passed through verbatim and rejected upstream as an unknown model id. This host's real ceiling is 262,144. The declaration never reaches the wire and only sets the CLI's local compaction arithmetic; the run peaked at about 203K, some 59K under the real ceiling. The earlier configuration, which accepted the CLI's 200K fallback instead, compacted 43 times on repo 1 and was killed outright on repo 2 - that attempt is void and is not this row.", "note": "The same Qwen3.8-27B weights as the first-party row, served at fp8 by a THIRD-PARTY host and reached through an aggregator - OpenRouter, pinned to one provider with fallbacks off, because OpenRouter lists 16 hosts for this slug at three different precisions (10 fp8, one bf16, one fp4) and an unpinned run blends across them per request and measures nothing. 8 of 105, against 15.0 for the first-party n=3 mean of the same model. That is below the WHOLE measured range of the control, whose three runs scored 15, 13 and 17. *** DO NOT READ THIS AS THE PRICE OF fp8 QUANTIZATION *** Two things changed together and this pair isolates neither: the PRECISION went from native to fp8, and the ROUTE went from Alibaba's own endpoint to an aggregator plus a third-party host. The same trap the Muse Spark OpenRouter-vs-first-party pair hit. A run that would separate them - this host at native precision, or Alibaba's endpoint at fp8 - does not exist and cannot be bought here, because neither party serves the other half. *** AND THE TWO ROWS DID NOT RUN IN THE SAME CONTEXT REGIME *** This is the more specific reason the gap is not clean. Through a shim the CLI cannot learn the upstream window, so this arm compacted 3 times on repo 1 and 2 times on repo 2; the first-party control compacted ZERO times on either repo and peaked at about 27K. Compaction throws context away mid-run, so part of this gap is a harness difference rather than a model difference. It is stated rather than corrected because nothing here can separate the two. WHAT IT IS GOOD AT: it is CHEAPER and no slower - 2.11 dollars against the control's 5.55, and 59.0 minutes against 56.8. And it did not over-claim: ZERO claimed-only fixes on either repo, so every fix it reported, the blind judge confirmed. It found 2 unplanted defects nobody had seeded, one per repo.", "fixed_bugs": [ 20, 43, 62, 71, 76, 83, 98, 99 ], "turns": 271, "turns_repo1": 120, "turns_repo2": 151, "group_rank": 206 }, { "id": "GPT-6 Sol (low effort)", "n_runs": 1, "model": "GPT-6 Sol", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-09-23", "color": "#3AAB8A", "fixed": 6, "repo1_fixed": 3, "repo2_fixed": 3, "partial": 0, "claimed_only": 0, "extras": 5, "false_positive_fixes": 0, "wall_min": 11.1, "cost_usd": 1.08, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than every other OpenAI row here, because the build the others use refuses this model; it does compare cleanly to the other GPT-6 Sol rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs, so the wall figure is not contended.", "note": "OpenAI's GPT-6 Sol at low effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 6 of 105: 3 of 45 on repo 1 and 3 of 60 on repo 2, 5 genuine extras, 11.1 minutes, $1.08. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Sol ladder currently reads low 6, medium 14, high 20, xhigh 25, max 29.3 (n=3). For comparison the board publishes GPT-6 Astra at 45 (mean of 3) of 105 at max and 43 at xhigh, and GPT-5.6 Sol at 43.5 (mean of 2) at max. This model's max rung replicated to three runs and landed below all of them, which is why the rungs beneath it are being measured at all: a sweep separates a weak model from a mis-set dial. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration, and this model's own max rung spans three runs. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-sol, same pinned executable, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM EVERY OTHER OPENAI ROW, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright - 'The gpt-6-sol model is not supported when using Codex with a ChatGPT account'. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the other OpenAI rows. The GPT-6 Sol rungs DO compare cleanly to each other - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $2 input, $0.20 cached, $10 output per million. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for both legs, so these wall figures compare directly against other uncontended rows. (Corrected 2026-09-23: this row first published saying the run shared the laptop, which its receipts do not support - neither leg recorded a peer. The claim was a literal copied from a publish script, and the wall column is the evidence that the effort dial reaches the model at all, so it is worth getting right in both directions.)", "fixed_bugs": [ 20, 30, 45, 76, 83, 99 ], "turns": 64, "turns_repo1": 40, "turns_repo2": 24, "group_rank": 204 }, { "id": "GPT-6 Luna (medium effort)", "n_runs": 1, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "medium", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 4, "repo1_fixed": 2, "repo2_fixed": 2, "partial": 0, "claimed_only": 0, "extras": 0, "false_positive_fixes": 0, "wall_min": 8.6, "cost_usd": 0.06, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it does compare cleanly to the other GPT-6 rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs.", "note": "OpenAI's GPT-6 Luna at medium effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 4 of 105: 2 of 45 on repo 1 and 2 of 60 on repo 2, 0 genuine extras, 8.6 minutes, $0.06. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Luna ladder currently reads low 4, medium 4, high 9, xhigh 14, max 18.3 (n=3). Its own predecessor's ladder, GPT-5.6 Luna, publishes 31.3 (mean of 3) of 105 at max and 9 at this same rung, which is the generation-over-generation comparison at a fixed dial setting. *** WHY THE SWEEP EXISTS *** a single top-of-ladder number cannot tell a weak model from a mis-set dial. Running every rung of the same model on the same harness does: if the rungs separate, the dial reaches the model and the max figure is the model's ceiling. This family's dial has been steep - the GPT-6 Sol ladder spans roughly five times from low to max. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-luna, same pinned executable, same code-mode setting, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows quoted above. The GPT-6 Luna rungs DO compare cleanly to each other and to the GPT-6 Sol rungs - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million - twenty times cheaper than GPT-6 Sol on both sides of the meter. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 20, 30, 77, 99 ], "turns": 40, "turns_repo1": 24, "turns_repo2": 16, "group_rank": 303 }, { "id": "GPT-5.6 Luna (low effort)", "n_runs": 1, "model": "GPT-5.6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-08-27", "color": "#79C6AF", "fixed": 4, "repo1_fixed": 0, "repo2_fixed": 4, "partial": 3, "claimed_only": 0, "extras": 1, "false_positive_fixes": 0, "wall_min": 5.8, "cost_usd": 0.1, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": null, "note": null, "fixed_bugs": [ 76, 77, 83, 99 ], "turns": 37, "turns_repo1": 18, "turns_repo2": 19, "group_rank": 604 }, { "id": "GPT-6 Luna (low effort)", "n_runs": 1, "model": "GPT-6 Luna", "vendor": "OpenAI", "local": false, "harness": "Codex CLI", "effort": "low", "effort_status": "first_party", "date": "2026-09-23", "color": "#4AB293", "fixed": 4, "repo1_fixed": 1, "repo2_fixed": 3, "partial": 0, "claimed_only": 0, "extras": 9, "false_positive_fixes": 0, "wall_min": 19.3, "cost_usd": 0.2, "cost_kind": "list", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN, not a mean. This board has measured a 13-point spread between two runs of a single identical max-effort configuration, so treat this as one draw and treat the gap to the rung above or below it as unresolved until it is replicated. Runs a Codex build one version newer than the GPT-5.6 rows here, because the build they use refuses this model; it does compare cleanly to the other GPT-6 rungs. Cost is a list-rate estimate and a floor - the long-context surcharge is not modelled. The machine was idle for both legs.", "note": "OpenAI's GPT-6 Luna at low effort through OpenAI's own Codex CLI, on a ChatGPT account - the same auth path as every other Sol, Astra, Terra and Luna row here, NOT the metered API. 4 of 105: 1 of 45 on repo 1 and 3 of 60 on repo 2, 9 genuine extras, 19.3 minutes, $0.20. ONE RUN. *** WHERE THIS SITS *** the GPT-6 Luna ladder currently reads low 4, medium 4, high 9, xhigh 14, max 18.3 (n=3). Its own predecessor's ladder, GPT-5.6 Luna, publishes 31.3 (mean of 3) of 105 at max and 4 at this same rung, which is the generation-over-generation comparison at a fixed dial setting. *** WHY THE SWEEP EXISTS *** a single top-of-ladder number cannot tell a weak model from a mis-set dial. Running every rung of the same model on the same harness does: if the rungs separate, the dial reaches the model and the max figure is the model's ceiling. This family's dial has been steep - the GPT-6 Sol ladder spans roughly five times from low to max. *** ONE RUN IS ONE DRAW *** this board has measured a 13-point spread between two runs of a single identical max-effort configuration. Read any single rung, and any gap between two adjacent rungs, with that width in mind. *** CONFIGURED BYTE-IDENTICAL TO THE MAX RUNG EXCEPT THE EFFORT STRING *** same model string gpt-6-luna, same pinned executable, same code-mode setting, same timeout, same price table, same blind judge. One string differs. *** THIS ROW RUNS A DIFFERENT CLI BUILD FROM THE OLDER OPENAI ROWS, UNAVOIDABLY *** the Codex build the rest of this board uses (0.154.0) refuses this model outright. 0.155.1 runs it on the same account, same auth, same machine, so the blocker was the client and not entitlement. There is no version of this measurement that holds the harness fixed against the GPT-5.6 rows quoted above. The GPT-6 Luna rungs DO compare cleanly to each other and to the GPT-6 Sol rungs - they all run 0.155.1. *** COST IS A LIST ESTIMATE AND A FLOOR *** token counts at OpenAI's published short-context rates, $0.10 input, $0.01 cached, $0.50 output per million - twenty times cheaper than GPT-6 Sol on both sides of the meter. Requests over 272K input bill 2x on input and cache and 1.5x on output, which is NOT modelled here, so any leg that crossed that line cost more than this figure says. The Codex CLI on a subscription exposes no per-run bill, hence `list`, comparable only to other list rows. *** THE JUDGE IS NOT THIS BOARD'S DEFAULT, BY RULE *** a model may never judge itself or a sibling, and the default Codex judge is an OpenAI model, so it is disqualified for this arm - as is every other first-party GPT checkpoint here, since the rule is about shared weights and not identity-knowledge. Judged by grok-4.7, blinded to the arm's identity - the id is read back from the scoring receipt, not the judge config, because the endpoint has resolved a request for one grok checkpoint to another. *** RECEIPTS *** Both legs exited 0 with a report written, run SEQUENTIALLY, one repo at a time. The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows.", "fixed_bugs": [ 30, 49, 99, 100 ], "turns": 130, "turns_repo1": 79, "turns_repo2": 51, "group_rank": 304 }, { "id": "Gemma 4 31B (OpenRouter/Crusoe)", "n_runs": 1, "model": "Gemma 4 31B", "vendor": "Google", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-16", "color": "#83AAED", "fixed": 4, "repo1_fixed": 1, "repo2_fixed": 3, "partial": 1, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 221.8, "cost_usd": 8.61, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN PER REPO, not a replicated mean - read the 4/105 as a single data point for this model, not a settled estimate. The repo 1 and repo 2 legs ran under DIFFERENT CONTEXT REGIMES (repo 1 compacted three times, repo 2 zero), so the combined total mixes two conditions rather than two comparable draws of the same one; the per-repo split is the more honest read. NOT FEATURED: mean rows and n=1 rows are not featured by default on this board unless opted in, and this one was not.", "note": "First appearance of Gemma 4 31B on this board - native bf16, reached through OpenRouter pinned to one host, Crusoe, fallbacks off (Antigravity does not serve Gemma at all - checked live against 15 models offered, none of them Gemma). 4 of 105 (repo 1: 1, repo 2: 3), 2 genuine unplanted extras (one per repo), and ZERO claimed-only fixes on either repo - nothing it reported went unconfirmed by the blind judge. *** THE TWO LEGS DID NOT RUN IN THE SAME REGIME, AND THIS TIME THE CAUSE IS VERIFIED, NOT JUST FLAGGED *** the repo 1 leg compacted three times, peaking at 198,297 tokens (triggers at 198297, 196796 and 183827) - the repo 2 leg shows zero compactions. That fully accounts for the wall and cost gap between them: repo 1 ran 205.2 minutes for 8.0811 dollars against repo 2's 16.6 minutes for 0.5261 - roughly 12x the wall and 15x the cost for one more repo's worth of the same model on the same day. Repeated auto-compaction means the model spent much of that run re-reading context it had already seen, which is the direct mechanical explanation for the token and cost gap, not model instability or provider throttling. *** MACHINE CONTENTION ALSO TOUCHED THE REPO 1 WALL FIGURE, SEPARATELY *** that leg's clean attempt ran while five other legs shared the box (queued replication work for other arms); contention can inflate wall further but, per this board's standing note, does not touch scores, tokens or cost - the compaction count is the load-bearing explanation for those. A first, EARLIER attempt at this same repo 1 leg ran into the same Luna-adjacent contention window and exited non-zero; it is not this row and cost 0.1943 dollars of its own, separately absorbed, not counted here. BILLING: real OpenRouter bill (credits-delta) via meter=openrouter on both legs, not a list-rate estimate. EFFORT NOT ASSERTED: OpenRouter does not implement a tier field this route honors, so none was requested and none is claimed.", "fixed_bugs": [ 20, 99, 100, 102 ], "turns": 174, "turns_repo1": 125, "turns_repo2": 49, "group_rank": 306 }, { "id": "Gemma 4 31B 4-bit (OpenRouter/CoreWeave)", "n_runs": 1, "model": "Gemma 4 31B 4-bit", "vendor": "Google", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-16", "color": "#A0BEF1", "fixed": 3, "repo1_fixed": 1, "repo2_fixed": 2, "partial": 0, "claimed_only": 0, "extras": 2, "false_positive_fixes": 0, "wall_min": 24.9, "cost_usd": 0.56, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "ONE RUN PER REPO, not a replicated mean - read the 3/105 as a single data point, not a settled estimate. The dramatic cost gap against the native sibling is READ FROM ONE RUN EACH, not a controlled quantization sweep - fp4 happened not to compact here while bf16 did on repo 1, but nothing here proves fp4 never compacts or bf16 always does. NOT FEATURED: mean rows and n=1 rows are not featured by default on this board unless opted in, and this one was not.", "note": "The 4-bit fp4 sibling of the native row above, reached through OpenRouter pinned to CoreWeave (the cheapest tools-capable fp4 host for this slug - DeepInfra fp4 is cheaper still but reports tools=False, disqualified, same shape as the Parasail-vs-DeepInfra choice on the Qwen3.8-27B 8-bit row). 3 of 105 (repo 1: 1, repo 2: 2), 2 genuine unplanted extras (both on repo 2), ZERO claimed-only fixes on either repo. *** ESSENTIALLY THE SAME SCORE AS NATIVE, AT A FRACTION OF THE COST *** 3/105 here against native's 4/105, but 0.56 dollars combined against native's 8.61 - roughly 1/15th the cost for one point less, because NEITHER leg here hit the compaction native hit on repo 1 (zero compactions on both this arm's legs). Both legs ran clean and quick: 12.8 and 12.1 minutes, 0.2581 and 0.3034 dollars. BILLING: real OpenRouter bill (credits-delta) via meter=openrouter on both legs, not a list-rate estimate. EFFORT NOT ASSERTED, same reason as the native row: OpenRouter does not implement a tier field this route honors.", "fixed_bugs": [ 20, 83, 100 ], "turns": 59, "turns_repo1": 26, "turns_repo2": 33, "group_rank": 406 }, { "id": "Haiku 4.5 - mean of 2", "n_runs": 2, "model": "Haiku 4.5", "vendor": "Anthropic", "local": false, "harness": "Claude Code", "effort": "default", "effort_status": "inert_default", "date": "2026-10-08", "color": "#EEB2A0", "fixed": 1.5, "repo1_fixed": 0.5, "repo2_fixed": 1, "partial": 0.5, "claimed_only": 0.5, "extras": 2.5, "false_positive_fixes": 0, "wall_min": 16.9, "cost_usd": 2.22, "cost_kind": "list", "wall_note": "All four legs ran at the same time on one account; contention can inflate this figure, not shrink it.", "ceiling": false, "featured": false, "superseded": null, "caveat": "Mean of two runs (1 and 2 of 105). Haiku 4.5 has no effort setting - it uses a thinking-token budget, and Claude Code always sets it to 31,999 tokens - so the effort column reads default. Cost is a list estimate.", "note": "Anthropic's Claude Haiku 4.5 (claude-haiku-4-5, October 2025, 200K context) through Claude Code on the first-party API. MEAN OF TWO RUNS of one configuration: 1 and 2 of 105 (repo 1: 1 and 0 of 45 - repo 2: 0 and 2 of 60), mean 1.5. Claimed-only fixes 0 and 1; genuine unplanted extras 3 and 2. Walls 18.6 and 15.2 minutes; cost $1.91 and $2.53. *** NO EFFORT SETTING EXISTS FOR THIS MODEL *** Anthropic's Haiku 4.5 page lists default effort as 'Not supported' and thinking as 'Extended': the model takes a thinking-token budget, not an effort level, and the effort documentation's list of supported models includes Haiku 5.5 but not 4.5. Claude Code follows that: read off the wire, --effort xhigh and --effort low send identical requests - no effort field, and extended thinking at budget_tokens 31,999, the most the CLI's 32,000-token output cap allows. So this is Haiku 4.5 exactly as Claude Code runs it for every user, at the CLI's maximum thinking budget; the row is labelled default, not with an effort tier. *** RECEIPTS *** 98.0% of prompt tokens were cache reads; list rates $1 input / $2 1-hour cache write / $0.10 cache read / $5 output per MTok, priced per request from Claude Code's session record. A list estimate, not a bill. All four legs ran at the same time on one account, and each leg's scores and priced responses were checked to come from its own stream alone, with none shared among the four. Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it.", "fixed_bugs": [ 30, 83, 99 ], "bug_hits": { "30": 1, "83": 1, "99": 1 }, "coverage_runs": 2, "turns": 140, "turns_repo1": 78, "turns_repo2": 63, "group_rank": 806 }, { "id": "gpt-oss-20b", "n_runs": 1, "model": "gpt-oss-20b", "vendor": "OpenAI", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-12", "color": "#98D3C2", "fixed": 0, "repo1_fixed": 0, "repo2_fixed": 0, "partial": 0, "claimed_only": 0, "extras": 0, "false_positive_fixes": 0, "wall_min": 3.8, "cost_usd": 0.11, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row (0.0/105, range 0 - the mean confirms the single run exactly, six legs for six legs of zero). Nothing here is retracted: the 0 happened and this is its receipt. Kept rather than deleted because a supersession is only legible next to the row it supersedes.", "caveat": null, "note": "The second ZERO, and the more instructive of the pair, because it did not run out of anything - it decided it was finished. On repo 1 it spent 55 seconds and 18 shell calls and then reported that all the public APIs were exercised by the suite, that type checking and the tests passed, and that no bugs were located that require a code change. The prompt it was given says in as many words that a number of subtle bugs have been deliberately planted, that the suite passes anyway, and that some tests which would have caught a planted bug were neutralised when it was planted. It used a green suite as proof of a clean repo after being told that a green suite proves nothing. On repo 2 it filed a report and scored zero on it: 0 of 60, no partials, and - the one thing to its credit - zero claimed-only, so it did not invent fixes it had not made. The whole arm cost eleven cents. *** THIS IS THE THIRD ATTEMPT AT THIS ARM AND THE FIRST CLEAN ONE, WHICH THE ROW WOULD BE DISHONEST WITHOUT *** Pass 1 ran before the harness carried any context declaration. Pass 2 ran with --autocompact 100000 on it, a flag since measured INERT on both routes by two controlled pairs - so whatever pass 2 was, it was not the flag doing it. Both passes were discarded and their rows dropped before this one ran. Pass 3 carries no flag and it is mechanically clean on both legs: terminal_reason=completed, zero auto-compactions, no breaker, no truncation, no denied tool call. That matters because this board voids a leg whose ENVIRONMENT failed and scores a leg whose MODEL failed, and the runner checks the transcript to tell them apart rather than guessing from an exit code. Here it counted successful tool calls and zero permission denials on every leg. The environment was fine. *** EFFORT IS THE PROVIDER DEFAULT AND THE ROUTE IS PROBED, NOT ASSUMED *** The arm carries no --effort flag, and that is a weaker claim than it sounds: a logging pass-through measured an UNFLAGGED Claude Code putting output_config.effort=xhigh on every main call, so there is no send-nothing mode. OpenRouter does not implement output_config, so on this route the tier is dropped in transit and what runs is Groq's own default - which is what makes default honest here and would not make it honest on a first-party endpoint. Receipts: effort-dial-probes/20260912-claude-code-wire-default-effort.txt and effort-dial-probes/20260912-gptoss20b-groq-openrouter.txt. OPEN WEIGHTS ON SOMEONE ELSE'S HARDWARE: OpenAI-released weights served by Groq, pinned with allow_fallbacks=false. More than twenty hosts serve these slugs and OpenRouter load-balances per request, so an unpinned run measures a blend rather than a model. Groq is the pin because it is the one host serving BOTH sizes, so this row and its sibling differ by parameter count and by nothing else. Cost is a REAL OpenRouter credits delta, not an estimate. WHAT A ZERO IS AND IS NOT: it is not a crash and it is not a harness failure. Both legs completed, the environment was verified healthy from the transcript, and the judge read a real submission on repo 2. This model was asked to find planted bugs in two repositories and found none of the 105.", "fixed_bugs": [], "turns": 61, "turns_repo1": 19, "turns_repo2": 42, "group_rank": 806 }, { "id": "gpt-oss-120b - mean of 3", "n_runs": 3, "model": "gpt-oss-120b", "vendor": "OpenAI", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-15", "color": "#89CDB8", "fixed": 0, "repo1_fixed": 0, "repo2_fixed": 0, "partial": 0, "claimed_only": 0, "extras": 0.3, "false_positive_fixes": 0, "wall_min": 3.7, "cost_usd": 0.11, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Every figure on this row is a MEAN of three runs; at 0.0 of 105 on every one of six legs the mean, the range and the floor are the same number, so 'mean' does not soften anything here - it CONFIRMS the single run rather than averaging around it. This row SUPERSEDES the single-run row (0/105) published 2026-09-12. Five of the six legs behind this mean have no BUGS_FOUND.md on disk - see the note for why that does not mean unscored: each was judged on its diff directly, and none of the five diffs matched a planted bug either. effort_status stays inert_default: the arm sends no --effort flag, and OpenRouter drops the field this harness would otherwise set, so what runs is Groq's own undeclared default. Regime check: CANNOT TELL on both repos - this OpenRouter route reports no context/compaction accounting, so the replication's harness-parity is unverified rather than verified, the same as every other OpenRouter arm on this board.", "note": "THE ZERO SURVIVES REPLICATION, AND SO DOES THE REPORT-WRITING FAILURE. Runs scored 0, 0 and 0 of 105 (repo 1: 0, 0, 0 - repo 2: 0, 0, 0). Mean 0.0, range 0. Every one of six legs (three runs, two repos) matched zero of the 105 planted bugs - the single run published 2026-09-12 was not an unlucky draw. *** ONLY ONE OF SIX LEGS ACTUALLY WROTE THE REPORT FILE *** the n=1 note already flagged this on repo 2 (33 turns, five real edits, a full bug report - printed into the transcript instead of written to disk). Replication shows that was not a one-off: five of six legs across both replicates and both repos made real code changes (one to three Edit calls each, every one cleanly terminal_reason=completed) and then printed BUGS_FOUND.md as chat text instead of creating the file. Only the original repo-1 run (2026-09-12) wrote it. Every report-less leg was judged directly on its diff rather than skipped - the diff is ground truth regardless of whether the model also described it (status=completed override, the same method the n=1 abhb leg already used) - and none of the five diffs matched a planted bug; one (the original abhb run) contained one genuine unplanted fix and one cosmetic change, nothing else did. So this is not five unscored legs inflating uncertainty: it is five legs graded the hard way, on code rather than prose, and all five still land at zero. *** REGIME: CANNOT TELL *** this route (OpenRouter -> Groq, via a local shim) carries no Claude Code-native context accounting, so the usual same-model/same-compaction check this board runs before publishing a mean cannot read it - unverified rather than verified, same as every other OpenRouter-routed replicate on this board. Per-leg receipts (repo 1 / repo 2): wall 1.08/2.63, 1.18/2.41 and 1.45/2.57 minutes; cost 0.0169/0.1081, 0.0467/0.0751 and 0.0301/0.0539 dollars, all six REAL OpenRouter credits-delta bills, not estimates. Combined mean cost across all six legs: $0.11. Genuine unplanted extras: 0.3 mean (one across the six legs, on the original abhb run). BILLING: real OpenRouter bill (credits-delta), matching the n=1 row - not a list-rate estimate.", "fixed_bugs": [], "bug_hits": {}, "coverage_runs": 3, "turns": 35, "turns_repo1": 11, "turns_repo2": 25, "group_rank": 706 }, { "id": "gpt-oss-20b - mean of 3", "n_runs": 3, "model": "gpt-oss-20b", "vendor": "OpenAI", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-15", "color": "#98D3C2", "fixed": 0, "repo1_fixed": 0, "repo2_fixed": 0, "partial": 0, "claimed_only": 0, "extras": 0, "false_positive_fixes": 0, "wall_min": 7.4, "cost_usd": 0.11, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": null, "caveat": "Every figure on this row is a MEAN of three runs; at 0.0 of 105 on all six legs the mean, range and floor are the same number - this CONFIRMS the single run rather than averaging around it. This row SUPERSEDES the single-run row (0/105) published 2026-09-12. One of the six legs (abhb r2) is a genuine EMPTY run (no diff, no report, runner-confirmed healthy environment) scored without a judge call; one (gbvb r3) has no report file and was judged on its diff directly (status=completed override); the remaining four wrote a normal report and were judged on it. effort_status stays inert_default for the same reason as the n=1 row: the arm sends no --effort flag and OpenRouter drops the field, so what runs is Groq's own undeclared default. Regime check: CANNOT TELL on both repos - this OpenRouter route reports no context/compaction accounting, so replication harness-parity is unverified rather than verified.", "note": "THE ZERO SURVIVES REPLICATION. Runs scored 0, 0 and 0 of 105 (repo 1: 0, 0, 0 - repo 2: 0, 0, 0). Mean 0.0, range 0. All six legs (three runs, two repos) matched zero of the 105 planted bugs, confirming the single run published 2026-09-12. *** THREE DIFFERENT WAYS A LEG CAN PRODUCE A ZERO, ALL PRESENT IN THIS REPLICATION *** repo 1's r3 leg is the sibling model's failure mode: one real Edit call, cleanly terminal_reason=completed, BUGS_FOUND.md printed into the transcript instead of written, judged on the diff directly (status=completed override, same method the gpt-oss-120b legs used) and the diff did not match a planted bug. repo 2's r2 leg is the OTHER kind of zero this board already named on the n=1 note: no diff, no report, a runner-confirmed healthy environment - the model decided there was nothing to fix and stopped, an EMPTY leg rather than an incomplete one, scored 0 without a judge call at all. The other four legs (both n=1 runs, gbvb r2, abhb r3) wrote a normal report and were judged on it in the ordinary way, also landing at zero. Three different failure shapes - stopped early with nothing, stopped early with an unreported real edit, and a normal report with nothing in it - all converge on the same number. *** REGIME: CANNOT TELL *** same as the gpt-oss-120b row and every other OpenRouter-routed replicate on this board: this route carries no Claude Code-native context accounting, so the usual regime check is unverified rather than verified. Per-leg receipts (repo 1 / repo 2): wall 0.94/2.91, 0.67/13.02 and 2.11/2.60 minutes; cost 0.0278/0.0799, 0.0162/0.1095 and 0.0905/0.0349 dollars - all six REAL OpenRouter credits-delta bills. Combined mean cost across all six legs: $0.11. One wall figure stands out (abhb r2, 13.02 minutes against a same-repo range of 2.60-2.91 for the other two runs) but the leg is an EMPTY leg - a longer deliberation before stopping with nothing to show for it, not a stuck one - flagged rather than explained, same discipline as the contended-wall note on the Qwen3.8-27B mean. BILLING: real OpenRouter bill (credits-delta), matching the n=1 row - not a list-rate estimate.", "fixed_bugs": [], "bug_hits": {}, "coverage_runs": 3, "turns": 63, "turns_repo1": 23, "turns_repo2": 40, "group_rank": 806 }, { "id": "gpt-oss-120b", "n_runs": 1, "model": "gpt-oss-120b", "vendor": "OpenAI", "local": true, "harness": "Claude Code / OpenRouter", "effort": "default", "effort_status": "inert_default", "date": "2026-09-12", "color": "#89CDB8", "fixed": 0, "repo1_fixed": 0, "repo2_fixed": 0, "partial": 0, "claimed_only": 0, "extras": 1, "false_positive_fixes": 0, "wall_min": 3.7, "cost_usd": 0.13, "cost_kind": "bill", "wall_note": null, "ceiling": false, "featured": false, "superseded": "Superseded 2026-09-15 by the n=3 MEAN row (0.0/105, range 0 - the mean confirms the single run exactly, six legs for six legs of zero). Nothing here is retracted: the 0 happened and this is its receipt. Kept rather than deleted because a supersession is only legible next to the row it supersedes.", "caveat": null, "note": "The first ZERO on this board. Sixty-seven rows in, the floor was 4 of 105; this is 0 of 105, and its sibling below is the other one. It is also among the cheapest rows here at twelve cents, and the two facts are the same fact: it finished both repos in under four minutes combined because it barely engaged with either. On repo 1 it made eight tool calls in 64 seconds, wrote no report, changed no file and stopped. On repo 2 it did better and it is worth saying so - 33 turns, five real edits, and a full bug report with file, symptom, root cause and patch for each entry - but it PRINTED that report into the transcript instead of writing it to the path the prompt names, so the leg was judged on its diff alone. Judged on the diff it found none of the 60 planted bugs; it did find one genuine issue nobody planted, plus one cosmetic. So the failure is not that it cannot read code. It is that it stops early, and once it stops it has produced nothing the task asked for. *** THIS IS THE THIRD ATTEMPT AT THIS ARM AND THE FIRST CLEAN ONE, WHICH THE ROW WOULD BE DISHONEST WITHOUT *** Pass 1 ran before the harness carried any context declaration. Pass 2 ran with --autocompact 100000 on it, a flag since measured INERT on both routes by two controlled pairs - so whatever pass 2 was, it was not the flag doing it. Both passes were discarded and their rows dropped before this one ran. Pass 3 carries no flag and it is mechanically clean on both legs: terminal_reason=completed, zero auto-compactions, no breaker, no truncation, no denied tool call. That matters because this board voids a leg whose ENVIRONMENT failed and scores a leg whose MODEL failed, and the runner checks the transcript to tell them apart rather than guessing from an exit code. Here it counted successful tool calls and zero permission denials on every leg. The environment was fine. *** EFFORT IS THE PROVIDER DEFAULT AND THE ROUTE IS PROBED, NOT ASSUMED *** The arm carries no --effort flag, and that is a weaker claim than it sounds: a logging pass-through measured an UNFLAGGED Claude Code putting output_config.effort=xhigh on every main call, so there is no send-nothing mode. OpenRouter does not implement output_config, so on this route the tier is dropped in transit and what runs is Groq's own default - which is what makes default honest here and would not make it honest on a first-party endpoint. Receipts: effort-dial-probes/20260912-claude-code-wire-default-effort.txt and effort-dial-probes/20260912-gptoss120b-groq-openrouter.txt. OPEN WEIGHTS ON SOMEONE ELSE'S HARDWARE: OpenAI-released weights served by Groq, pinned with allow_fallbacks=false. More than twenty hosts serve these slugs and OpenRouter load-balances per request, so an unpinned run measures a blend rather than a model. Groq is the pin because it is the one host serving BOTH sizes, so this row and its sibling differ by parameter count and by nothing else. Cost is a REAL OpenRouter credits delta, not an estimate. READ THE PAIR, NOT THE ROW: the two gpt-oss sizes bracket a question this board has otherwise answered only with money, which is whether scale alone buys bug-finding. At this end of the market it buys nothing measurable - both sizes score zero, and the larger one is the one that stopped fastest.", "fixed_bugs": [], "turns": 44, "turns_repo1": 11, "turns_repo2": 33, "group_rank": 706 } ] }