The best AI coding agent fixes real bugs on unfamiliar code 38.8% of the time. Here's the math before your estimator builds anything with it

A new benchmark that tests AI coding agents on real, private company codebases — not public test sets the models may have already seen — found that even the best-performing agent resolves less than 40% of real bugs on the first try. That number matters if anyone at your company has asked an AI coding tool to build a bid tracker, a scope-gap checklist, or a takeoff spreadsheet macro, because it's the same kind of code: unfamiliar, specific to your business, and untested by the model beforehand.

What is the Real-SWE benchmark, and why is it different?

Most AI coding benchmarks run on public GitHub repositories, which frontier models have likely trained on — so a high score can just mean the model has seen the answer before. Real-SWE, built by YC-backed Specific Labs, instead licenses real production codebases from real companies — a consumer app with 200,000-plus users, a fintech platform that processes over 100,000 bank statements, enterprise sales tools — and gives each AI agent the same task and context an engineer on that team would have had. A human grader then checks whether the fix actually works. The median task touches 11 files and the median instruction runs about 1,742 characters — closer to "fix this thing our estimator flagged" than a tidy textbook problem.

That's the honest version of what happens when a GC or a trade sub points an AI coding agent at its own homegrown spreadsheet macro or internal tracker instead of a demo app: the agent has never seen this code before, and it has to work inside the mess that's actually there.

How well did the top agents actually do?

Scores are pass@1, averaged over eight runs per task. The takeaway isn't that Fable 5.1 "wins" — it's that the best agent tested still gets the fix wrong on unfamiliar code more often than it gets it right, and a failed attempt still costs money. Because Fable 5.1 fails on 6 out of 10 tasks, its true cost per completed fix (~$17.94) runs more than double its per-attempt price. Flash is cheaper per try and per completed fix, but it also finishes fewer jobs, so a shop weighing "cheap model, more retries" against "pricier model, fewer retries" needs to actually track completions, not sticker price.

Why does the failure mode matter more than the headline score?

Real-SWE also grouped failures by type, and the pattern differs by model. Grok 4.6's failures were dominated by missed requirements — 67.2% of its misses came from not doing what was actually asked. Gemini 3.8 Flash's failures were mostly integration errors (49.1%) — code that ran but broke something else it touched. GPT-5.6 Sol's failures leaned toward unverified assumptions about how the existing system worked (43.3%). Fable 5.1 and GPT-6 Astra spread their failures more evenly across categories, with no single failure type above 37%.

That's a practical distinction for anyone reviewing AI-generated code they didn't write: a model that mostly misses requirements is caught by re-reading the original ask against the diff. A model that mostly breaks other parts of the code is caught by actually running the tool against real data before trusting it. One review checklist doesn't cover both failure patterns — the check has to match the tool.

Should a mid-size GC let an agent build its own tools?

Yes, with a review gate, and no, not unsupervised on anything touching live data. If an estimator or PM is using a coding agent to build a submittal tracker, a bid-leveling sheet, or a scope-gap checker — the kind of DIY tool this Brief has flagged before — treat the first output as a draft, not a deliverable. Budget for the agent to be wrong on the first attempt more often than it's right, run the tool against a copy of real project data before it touches the original, and have someone who understands the underlying spreadsheet or workflow check the diff, not just whether the tool "runs." The benchmark's core finding — best-in-field still under 40% on real code — isn't a reason to avoid these tools. It's a reason to keep a human between the agent's output and anything your team actually relies on.