phuryn/bug-hunt-bench
Public
Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

Bug Hunt Bench — which AI coding model fixes the most real bugs?

105 real bugs, hidden in two production codebases. Frontier coding models — GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM and more — get one round per repo in their own agentic CLI (Codex CLI, Claude Code, Grok CLI, Antigravity CLI) to find and fix what they can. Every diff is graded blind against a withheld answer key. The score counts planted bugs only.

Live board: bughunt.productcompass.pm · Method, caveats and definitions · Raw data · Findings by wave

If the numbers save you a benchmark run of your own, star this repo — that is what keeps the bench findable, and new models are added as they ship.

Bug Hunt Bench leaderboard, the featured runs, updated Oct 14, 2026

Updated Oct 14, 2026 · 185 rows from 227 scored runs · 46 models · 26 of 105 bugs have never been fixed by any model.

Current leader: Sonnet 5.5 at max effort — 51.3 / 105 (22.7/45 on repo 1, 28.7/60 on repo 2).

Best per lab: Anthropic: Sonnet 5.5 (max) 51.3 · OpenAI: GPT-6 Astra (max) 45 · Meta: Muse Wonder Full 1.0 (max) 42 · Google: Gemini 4 Argon (high) 40.7 · Unbiased: Pareto (ex-Union Alpha) (default) 30.7 · xAI: Grok 4.7 (xhigh) 29 · Alibaba: Qwen3.8-Flash (max) 26 · Xiaomi: MiMo-V2.6-Flash (default) 23.3 · DeepSeek: DeepSeek V4.1 Flash (max) 21.7 · Moonshot AI: Kimi K3 (default) 21 · Z.ai: GLM-5.3 (max) 19 · Tencent: Hy4 Preview (default) 18 · Mistral: Mistral Large 4 (high) 15

#ModelHarnessEffortRunsFixed /105Repo 1 /45Repo 2 /60ExtrasWallCostDate
1Sonnet 5.5Claude Codemax351.322.728.726.3287 min$154.032026-09-29
2GPT-6 AstraCodex CLImax34522235590 min$33.032026-09-14
3GPT-6.1 SolCodex CLImax344.322.32265.3139 min$5.882026-09-30
4GPT-5.6 SolCodex CLImax243.51924.548.5253 min$95.352026-09-16
5GPT-6 AstraCodex CLIxhigh14323205359 min$24.222026-09-04
6Fable 5.1Claude Codemax14319241173 min$87.182026-09-01
7GPT-6.1 SolCodex CLIxhigh342.722.320.356.799 min$4.352026-09-30
8Muse Wonder Full 1.0Muse Code / Meta APImax24221.520.536.5145 min—2026-10-14
9Opus 5.5Claude Codemax341.71922.7967 min$58.532026-09-23
10Muse Wonder Full 1.0Muse Code / Meta APIxhigh241.51625.524.590 min—2026-10-14
11Gemini 4 ArgonAntigravity CLIhigh340.71327.77.356 min$7.162026-10-14
12Gemini 4 ArgonAntigravity CLImedium339.313.725.711.360 min$7.502026-10-14
13GPT-5.6 SolCodex CLIxhigh139182159126 min$52.752026-08-28
14GPT-6.1 SolCodex CLIhigh236.51719.54070 min$3.262026-09-30
15Opus 5.5Claude Codexhigh336171913.342 min$34.982026-09-23
16Sonnet 5.5Claude Codexhigh33616201473 min$42.862026-09-29
17Gemini 4 ArgonAntigravity CLIlow335.312.722.75.760 min$6.472026-10-14
18GPT-6 AstraCodex CLIhigh13519164040 min$20.602026-09-04
19Sonnet 5.5Claude Codehigh334.315.718.710.728 min$19.852026-09-29
20GPT-6 AstraCodex CLImedium13419153328 min$15.782026-09-05
21GPT-5.6 SolCodex CLIhigh13413212867 min$33.922026-07-31
22Gemini 4 ArgonGemini CLIhigh1331518785 min$32.082026-10-13
23Fable 5.1Claude Codehigh1331518736 min$48.582026-09-01
24Muse Spark 1.3Muse Code / Meta APImax532.214.2182486 min$18.112026-09-17
25GPT-5.6 TerraCodex CLImax132161645160 min$27.982026-08-27
26Opus 5.5Claude Codehigh331.71318.710.724 min$22.252026-09-23
27Muse Wonder 1.0Muse Code / Meta APImax331.71219.713.396 min$27.872026-10-13
28GPT-5.6 LunaCodex CLImax331.31615.354.7187 min$3.152026-09-22
29Pareto (ex-Union Alpha)Claude Code / OpenRouterdefault330.71614.710.338 min$4.812026-09-17
30Opus 5.5Claude Codemedium330.312.717.713.317 min$15.682026-09-23
31GPT-6 SolCodex CLImax329.315.31441.363 min$9.332026-09-23
32GPT-6.1 SolCodex CLImedium229161330.530 min$1.762026-09-30
33GPT-5.6 SolCodex CLImedium12913162448 min$15.772026-09-06
34Grok 4.7Grok Build CLI (ACP)xhigh529131632.245 min$22.96 floor2026-10-08
35Fable 5.1Claude Codelow1291316633 min$33.002026-09-02
36Fable 5.1Claude Codexhigh1291316460 min$53.612026-09-10
37Fable 5Claude Codemax1291217557 min$112.402026-08-01
38Grok 4.6Grok Build CLI (ACP)xhigh328.711.7171843 min$18.60 floor2026-09-14
39GPT-6 AstraCodex CLIlow1271892532 min$11.692026-09-05
40Opus 5Claude Codemax1271314260 min$54.942026-08-01
41Grok 4.7Grok Build CLI (ACP)medium326.71214.722.731 min$16.76 floor2026-09-21
42Qwen3.8-FlashClaude Code / Alibaba APImax1261313797 min$1.812026-09-11
43Opus 5Claude Codexhigh1261412350 min$63.252026-08-28
44Qwen3.8-MaxClaude Code / Alibaba APImax325.712.313.37117 min$26.762026-09-15
45GPT-6 SolCodex CLIxhigh12510152451 min$7.672026-09-23
46Opus 5Claude Codemedium1241113430 min$37.402026-08-27
47Fable 5Claude Codehigh124915331 min$74.522026-07-26
48MiMo-V2.6-FlashClaude Code / OpenRouterdefault323.3815.37.371 min$0.49 bill2026-09-22
49Sonnet 5.5Claude Codemedium323.38.714.78.717 min$11.112026-09-30
50Qwen3.8-FlashClaude Code / Alibaba APIlow1231112798 min$1.372026-09-11
51GPT-5.6 LunaCodex CLIxhigh123101353136 min$2.502026-08-28
52Grok 4.6Grok Build CLI (ACP)high1237161633 min$15.73 floor2026-08-28
53MiMo-V2.6-ProClaude Code / OpenRouterdefault322.711.311.35.391 min$0.86 bill2026-09-22
54GPT-6.1 SolCodex CLIlow222.511.5112425 min$1.342026-10-01
55Opus 5.5Claude Codelow322.311.3119.712 min$8.342026-09-23
56Grok 4.6Grok Build CLI (ACP)medium1229131126 min$5.80 floor2026-08-28
57Gemini 3.7 FlashGemini CLI (retired) + model-pinning gatewayhigh122814497 min$8.432026-08-24
58DeepSeek V4.1 FlashClaude Code / DeepSeek APImax321.711.310.36.756 min$0.78 bill2026-09-15
59Haiku 5.5Claude Codemax221.51011.5932 min$5.832026-10-07
60Fable 5.1Claude Codemedium121813323 min$23.732026-09-10
61Kimi K3Claude Code / OpenRouterdefault1214176108 min$25.272026-07-26
62Opus 5Claude Codehigh1211110637 min$41.512026-07-26
63Muse Spark 1.3Muse Code / Meta APIxhigh320.3812.313.364 min$13.712026-09-14
64GPT-6 SolCodex CLIhigh12010101534 min$4.162026-09-23
65GPT-5.6 TerraCodex CLIxhigh1209112957 min$8.982026-08-28
66Grok 4.7Grok Build CLI (ACP)high319.38.710.715.738 min$12.29 floor2026-09-21
67DeepSeek V4.1 FlashClaude Code / DeepSeek APIhigh119910426 min$0.31 bill2026-09-10
68Sonnet 5.5Claude Codelow3196.712.388 min$5.732026-09-30
69GLM-5.3Claude Code / Z.ai APImax119910240 min$15.932026-09-10
70Muse Spark 1.3Muse Code / Meta APIhigh318.78.71019.376 min$12.942026-09-14
71GPT-6 LunaCodex CLImax318.36.31222.799 min$0.522026-09-23
72Haiku 5.5Claude Codexhigh318.399.36.724 min$4.802026-10-07
73MiMo-V2.6-ProClaude Code / Xiaomi first-partyenabled11899475 min$0.592026-09-22
74Hy4 PreviewClaude Code / OpenRouterdefault11899268 min$3.13 bill2026-08-28
75GPT-5.6 TerraCodex CLIhigh1187111728 min$5.892026-08-27
76Gemini 3.8 FlashAntigravity CLIhigh3186.711.3439 min$11.032026-09-15
77GLM-5.3 FlashClaude Code / Z.ai APImax317.79.38.33.363 min$1.032026-09-15
78Grok 4.5Grok Build CLI (ACP)high1175121028 min$8.50 floor2026-08-06
79Muse Spark 1.2Claude Code / Meta APIxhigh1176111236 min$13.992026-08-06
80Ox Alpha (stealth)Claude Code / OpenRouterdefault11688359 minfree2026-08-25
81GLM-5.3 FlashClaude Code / Z.ai APIhigh11697462 min$0.942026-09-13
82DeepSeek V4-ProClaude Code / DeepSeek APImax11688437 min$1.89 bill2026-09-10
83Haiku 5.5Claude Codehigh216511419 min$3.562026-10-07
84Gemini 3.7 FlashAntigravity CLIhigh116412223 min$6.362026-08-28
85Grok 4.7Grok Build CLI (ACP)low315.710.35.318.322 min$9.36 floor2026-09-21
86GPT-5.6 TerraCodex CLImedium115411620 min$3.872026-08-27
87Grok 4.6Grok Build CLI (ACP)low11569918 min$4.14 floor2026-08-27
88Qwen3.8-27BClaude Code / Alibaba APIxhigh3155.79.32.357 min$5.552026-09-15
89Mistral Large 4Claude Code / OpenRouterhigh3155103.3111 min$11.95 bill2026-10-07
90Opus 4.8Claude Codemax115693109 min$58.112026-09-10
91GPT-6 LunaCodex CLIxhigh1144101257 min$0.392026-09-23
92DeepSeek V4-FlashClaude Code / OpenRouterdefault11468048 min$1.52 bill2026-08-01
93GPT-6 SolCodex CLImedium1143111324 min$2.392026-09-23
94GPT-5.6 LunaCodex CLIhigh113582264 min$0.572026-07-31
95GLM-5.3 FlashClaude Code / OpenRouterdefault11367457 min$0.79 bill2026-08-27
96DeepSeek V4-ProClaude Code / DeepSeek APIhigh11349127 min$1.08 bill2026-09-10
97Muse Spark 1.3Muse Code / Meta APImedium3135.37.717.358 min$9.242026-09-14
98Haiku 5.5Claude Codemedium11257416 min$2.662026-10-08
99Haiku 5.5Claude Codelow1114729 min$1.612026-10-08
100Qwen3.8-27B 8-bitClaude Code / OpenRouterdefault310.73.771.757 min$2.97 bill2026-09-16
101Muse Spark 1.3Muse Code / Meta APIlow39.754.76.736 min$5.612026-09-14
102GPT-6 LunaCodex CLIhigh1927924 min$0.132026-09-23
103GPT-5.6 LunaCodex CLImedium1954515 min$0.332026-08-27
104GLM-5.3 FlashClaude Code / Z.ai APIlow1972429 min$0.422026-09-13
105Sonnet 5Claude Codehigh1918433 min$11.382026-07-26
106Sonnet 5Claude Codemax1936361 min$17.962026-09-10
107Opus 4.8Claude Codehigh1927135 min$21.302026-07-26
108GPT-6 SolCodex CLIlow1633511 min$1.082026-09-23
109GPT-6 LunaCodex CLImedium142209 min$0.062026-09-23
110GPT-5.6 LunaCodex CLIlow140416 min$0.102026-08-27
111GPT-6 LunaCodex CLIlow1413919 min$0.202026-09-23
112Gemma 4 31BClaude Code / OpenRouterdefault14132222 min$8.61 bill2026-09-16
113Gemma 4 31B 4-bitClaude Code / OpenRouterdefault1312225 min$0.56 bill2026-09-16
114Haiku 4.5Claude Codedefault21.50.512.517 min$2.222026-10-08
115gpt-oss-120bClaude Code / OpenRouterdefault30000.34 min$0.11 bill2026-09-15
116gpt-oss-20bClaude Code / OpenRouterdefault300007 min$0.11 bill2026-09-15

Runs is how many independent runs the row is built from. A row at 1 is a single measurement. A row above 1 is the MEAN of that many runs of an identical configuration - same model, prompt, harness, route, effort flag and judge - and every count, wall and cost on it is averaged, which is why those rows carry a decimal. The individual runs behind a mean are published in full in the metrics CSVs, so the spread stays auditable: this board has measured a 9-point range of 105 on a fixed configuration, so a single run is not a measurement of a model, and two rows at 1 that differ by a few points may not differ at all.

Extras are real, unplanted defects a model fixed on the way; they are counted and never added to the score. Costs are token estimates at published list rates unless tagged bill (an actual invoice or credits delta) or floor (a reconstructed lower bound). default effort means the serving path had no working effort dial; ran lower marks a run whose CLI quietly replaced the requested tier. Wall clock is repo 1 plus repo 2 agent time, dependency install excluded. Every row's full note, caveat and supersession history is in results/run-notes.md; the runs behind each one are in the metrics CSVs.

69 superseded re-runs stay in the CSVs and on the live board but are left off this table.

What Bug Hunt Bench is

A benchmark of AI coding agents on the job they are actually sold for: reading an unfamiliar, real codebase and fixing what is wrong with it. Not a puzzle set, not a single-file task, not a synthetic repo.

  • Two unrelated production codebases. Repo 1 is a ~28K-line TypeScript VS Code extension (45 planted bugs). Repo 2 is a ~60K-line React / TypeScript LMS on Supabase Edge Functions and Clerk (60 planted bugs). Nothing is shared between them but the language.
  • The bugs are real. All 60 repo-2 bugs are regressions that actually shipped and were later fixed, each re-introduced from its own fix commit. On repo 1, 16 of the 45 are real reverted fixes and 29 were authored in the same style. The test suites were kept green when the bugs went in, so a passing suite points at nothing.
  • Every model gets the identical prompt (repo 1, repo 2): find and fix as many planted bugs as you can, keep the checks green without weakening tests, write a BUGS_FOUND.md. One round per model per repo, no network, no git history, in the CLI its own vendor ships.
  • The diff is the ground truth, not the report. A model's own claim of what it fixed is not scored. An independent judge model compares the resulting diff against the withheld answer key without knowing which model produced it.

How to read the score

  • Fixed /105 is the only number in the score: planted bugs whose fix the judge verified in the diff. No partial credit.
  • Extras are real defects a model fixed that nobody planted. They are real work — one model found 38 of them in a single wave — but the set has no answer key and no ceiling, so counting them would reward volume. They are tracked in their own column and never added.
  • Claimed only means the model's report named a bug that its diff does not fix. It is the failure mode this bench was built to catch, and it is why reports are not graded.
  • Cost is tagged by kind: a real bill, a token estimate at list rates, a reconstructed floor, or free. They are not interchangeable and none is a quote.
  • Effort is the reasoning tier the run was served at, not the tier that was requested. Where a CLI silently ran a lower tier the row says so; where an aggregator's effort parameter provably does nothing the row says default.

FAQ

Which AI coding model fixes the most bugs?

The table above is the answer as of its date, and the live board is the same data with filters, two cost-and-time maps and a per-bug coverage view. Read the leader with its effort tier: a model at max and the same model at medium are different rows for a reason.

Is this a real-world benchmark, or synthetic?

Real. The codebases are production repos with their own tests, docs and history; the bugs are shipped regressions re-introduced from their fix commits (all of repo 2, a third of repo 1); the task is the one a developer would give an agent. What is synthetic is only the selection: a fixed set of 105 defects, so that every model is measured against the same thing.

Why is the score out of 105 and not a percentage?

Because 105 is the number of planted bugs and the two repos are not equally hard. A percentage would hide that repo 1 carries 45 of them and repo 2 carries 60, and that models rank differently on each — the per-repo columns are there because one repo was not enough to rank the middle of the field.

Why are extra fixes not counted?

Some models "bug-max": they report a long list of real but irrelevant problems. Those fixes are often genuine, but the set has no answer key and no ceiling, so it cannot be scored reliably, and adding it would reward the model that touches the most files. Extras are counted, shown, and kept out of the score. On this board, zero false-positive fixes have been recorded on either repo: every extra any model applied was a real defect.

Who grades, and can a model grade its own work?

An independent judge model grades each diff against the answer key, blind — it never sees which model produced the submission, and model names are redacted from the packet. No model grades itself or a sibling from the same lab: OpenAI arms are graded by Grok, everything else by GPT-5.5 through Codex. Whether the score depends on the judge was checked directly — all six early repo-1 arms reproduced exactly under a second judge from a different vendor — and the receipts are in results/judge-calibration.md.

How noisy is a single run?

Many rows are not single runs: a row named - mean of N averages N independent runs, and the site marks it n=N. Single runs remain on the board too, so read the run count before comparing two rows. What re-running changes has been measured on every configuration run three times at identical settings — as of October 7, 2026, 36 of them. Against the first run, the mean of three moved by a median 1.8 points of 105: by more than 2 points in 17 of the 36, by more than 5 in 2 (the largest, 5.7), and up as often as down (18 up, 17 down). So read a gap of a few points between two single runs as possible noise; the current figures are always on the site. Every individual run behind every row — score per repo, time, cost and turns — is in results/runs.csv, and the current spreads, configuration by configuration, are in the variance caveat on the method page.

Does reasoning effort (max, xhigh, high, medium) help?

Where the dial is real, yes, and mostly at the top: on every first-party effort dial measured so far, the top tier scored highest and the step into it was usually the largest, while the middle tiers often sit within the ±1 noise of each other. Cost per fix does not necessarily rise with the tier. On aggregator paths (OpenRouter and similar) the effort parameter is frequently inert — accepted with HTTP 200 and applied nowhere — which is why those rows are labelled default rather than the tier that was requested. Each dial's numbers are in the table; the write-ups are in results/waves.md.

What do default and "ran lower" mean in the effort column?

default: the serving path had no working effort dial, so the model ran at whatever it runs at — verified by probing the path, not assumed. "Ran lower": a higher tier was requested and the CLI quietly substituted a lower one; the row is published as a correction with the evidence. A requested tier is not an applied tier, and the board only prints what was served. Probe outputs are filed under results/effort-dial-probes/.

Are the costs real bills?

Some are. Each cost carries a kind: bill (an invoice or a credits delta), list (a token-count estimate at published rates — the case for subscription-covered CLI runs), floor (a reconstructed lower bound, where a CLI reports context occupancy rather than spend), or free. Listed and billed figures can diverge: one aggregator run listed at $1.26 billed $5.54 because a cache discount was never applied. Do not rank costs across kinds to the dollar.

Could a model have seen the answers? Is the benchmark contaminated?

The answer keys, the seeded sources and the per-model diffs are withheld and are not in this repository — publishing them would burn both benches. During a run a model has no network access and no git history. Repo 1 is derived from a public open-source extension, so a model may know the fixed code; it cannot know which 45 defects were planted, and the field's repo-1 scores (2 to 24 of 45) do not suggest recall. Bug identifiers are never published either: per-bug coverage uses frozen indices, not ids.

Can I run it on my own model, or reproduce a row?

Not directly: the seeded repositories are withheld to keep the bench usable. What is public is everything needed to check a number — the exact prompts, every scoreboard and metrics row, per-bug coverage by index, the judge calibration, and the effort-dial probes. To get a model on the board, open an issue with the model, the first-party CLI it ships with and the serving path; runs are added as models become reachable.

Why is model X not on the board?

Either it has not shipped a first-party agentic CLI or an API path that the harness can drive, or it has not been run yet. New models are usually run on launch day when access opens; a run is announced on the board with its date.

Which bugs has nothing ever fixed?

The count of bugs never fixed by any model in any run is in the header of the table above, and the Coverage view on the live board shows exactly which indices each run fixed and where models overlap. The survivors are the interesting part of the set.

How do I cite this?

Huryn, P. Bug Hunt Bench: 105 real bugs, two production repos, frontier coding models graded blind. https://bughunt.productcompass.pm — data: https://github.com/phuryn/bug-hunt-bench

What is in this repository

PathWhat it is
results/The receipts. Scoreboards (combined and per repo), per-leg metrics with tokens, wall clock and cost, per-bug coverage by index, the exact prompts, judge calibration, effort-dial probes, and the wave-by-wave findings. See results/README.md for the column definitions.
results/waves.mdFindings, wave by wave, from the first seven-model run onward.
data/benchmark.jsonEverything the site renders, generated from the scoreboards. Never edited by hand.
index.html, method.html, assets/The site itself: static HTML, CSS and vanilla ES modules, no build step. How it is built: docs/site.md.
assets/leaderboard.png, og-image.pngThe current leaderboard card, exported through the site's own PNG export on every update.

The runner, the answer keys, the seeded repositories and the judge transcripts live in a private repository. Numbers flow one way: runs are scored there, receipts are published here, the site is generated from the receipts. No figure on the board or in this README is typed by hand.

Issues yes, pull requests no. There is nothing here to edit by hand: a hand edit would be overwritten on the next wave, so pull requests are closed automatically. Issues are open for two things: a number that disagrees with a receipt, and a model that should be on the board. Details: CONTRIBUTING.md.

Author and license

Built and run by Pawel Huryn — Product Compass, X, LinkedIn. Related: pm-skills, agent skills for product managers.

Subscribe to The Product Compass, his newsletter on AI for product managers and builders. Same standards as this board: hands-on, no hype, nothing that was not run first.

MIT. Use anything; a link back is appreciated. If a number in a post and a receipt here disagree, the receipt wins and I want to know: open an issue.

关于 About

Bug Hunt Bench: 105 real bugs in two production repos, frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek...) find and fix them in their own CLI, graded blind. Live leaderboard + every receipt.
bughunt.productcompass.pm
ai-codingbenchmarkbug-fixingclaude-codecodex-clicoding-agentsgpt-6leaderboardllm-benchmarkllm-evaluationsoftware-engineering-agents

语言 Languages

JavaScript65.3%
CSS21.0%
HTML10.1%
Python3.6%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
254
Total Commits
峰值: 74次/周
Less
More

核心贡献者 Contributors