When 85% on SWE-bench Turns Into 58% Under Proof

Also available as a plain-text transcript page.

0:00Paige: Claude sits at eighty-five percent on SWE-bench Verified, the number the entire coding-agent industry points to, and says, "this model can fix real bugs." Take those same patches, the ones that passed every test the maintainers wrote, and set an adversarial auditor loose, hunting for a single input where they disagree with the maintainers' own fix. The score falls to fifty-eight.

0:20Eric: Fifty-eight. So more than a quarter of what the leaderboard called a win... didn't match the official fix?

0:27Paige: Didn't match it. And we need to be precise about "match," because it's carrying more weight in that sentence than it seems. SWE-bench is the yardstick behind almost every claim you hear, about how close coding agents are to being trustworthy. And this is the first time anyone's checked that yardstick, against something a test suite can't fake its way past.

0:47Eric: And the obvious read is: fine, tests are imperfect, and everybody already knows that. But if a patch passes every test the maintainers actually wrote, and still does the wrong thing somewhere, wouldn't one of those tests eventually catch it? That's kind of the working assumption behind the whole benchmark.

1:04Paige: It's a reasonable assumption, and it's wrong for an almost embarrassing reason. The median instance in SWE-bench Verified is decided by exactly one newly passing test, not a suite probing the function from every angle. It's one input, chosen by whichever maintainer happened to write a regression test, on the day they fixed the bug.

1:22Eric: So one test decides whether "resolved" really means resolved.

1:26Paige: For the median instance, yes. And a test is a single point: you check the function on that input, and infer the rest. A proof checks it against every input the specification allows, all at once, without running anything. It's a ten-word spelling quiz versus the whole dictionary. Passing the quiz is real evidence you can spell. It's just evidence about ten words, not about the language. So if a proof gives you the whole dictionary, why hasn't anyone done this for real code before? Because a real fix doesn't live in ten clean lines. It sits inside a codebase with a hundred thousand lines, and calls dozens of functions nobody is ever going to formalize. And the real requirement is a bug report, typed at some point by a frustrated user, not a specification.

2:07Eric: Right, and the existing formal-code benchmarks basically dodge that. They either hand the model a spec to start with, or stay at textbook scale, where there's no real dependency graph to worry about.

2:19Paige: Which is the gap the authors built something called Benchproofer to close. It's a pipeline that takes a real, already-fixed GitHub issue, and manufactures a formally verified twin of it: a written specification of what the fix should do, a reference version of the code proven to satisfy that spec, and the real Python patch that does the same thing, sitting right next to it.

2:39Eric: Two pieces I want defined before we go further: the specification, and this thing you keep calling an axiom. That sounds like the part that makes any of this possible, on a real codebase at all.

2:50Paige: It is. But before the axiom, look at what a specification is, because "formal spec" sounds cold, and this one isn't. There's a matplotlib bug in the corpus: draw a histogram a certain way, and the chart's bounding box gets computed wrong, so part of the plot gets cut off. The spec for the fix says: given a list of points, the returned box has to satisfy two things. Every point has to sit inside it, and for each of the four edges, some point has to touch it.

3:16Eric: Why the second half? "Every point's inside the box" sounds like the whole requirement.

3:22Paige: Because a box the size of the universe also satisfies, "every point is inside it." Without something touching every edge, the spec is technically true and completely useless. It's a warehouse, not shrink wrap. That one omission, stating what has to be included but never what mustn't be wasted, is most of what goes wrong later, when a model has to write its own spec instead of being handed one.

3:45Eric: So how do you write anything like that, for a fix that calls forty other functions you're never going to touch?

3:51Paige: You don't formalize them. You formalize only the code the patch actually changes, and for everything it calls but leaves alone, you write one short assumed fact instead of a proof. That's the axiom. Picture subcontracting a bolt for a structure you're building. You don't inspect the foundry. You get a written guarantee — "this bolt holds five hundred kilos" — and you design around that number alone.

4:14Eric: And if the real bolt holds a thousand, no harm done. You just built conservatively.

4:19Paige: Right — but if the guarantee says five hundred, and the real bolt snaps at three hundred, the whole structural calculation is worthless, even though the math on paper was flawless. That's why an axiom can understate what a function does, but it can never overstate it. So every axiom gets fuzzed against the real function, running inside that task's own environment, hunting for one input that breaks the guarantee. Find one, and the axiom gets weakened until it survives.

4:47Eric: That handles "assumed too much." What stops a model, or the pipeline itself, from writing a spec that's true but empty: your universe-sized box?

4:56Paige: Three checks stand between a spec and that universe-sized box, and they're mechanical on purpose. Every instance comes with the old, buggy version of the code, and that version has to fail the same spec. If the bug still verifies, the spec never saw the bug at all. Then they mutate the correct implementation, one operator at a time, and those mutants have to stop verifying too. And then adversarial models get turned loose, specifically to write a wrong implementation that verifies anyway.

5:24Eric: And when an attacker's wrong implementation does verify?

5:28Paige: Then it goes to a tiebreak, because a wrong implementation slipping through doesn't automatically mean the spec's broken. A good spec should admit correct programs written differently from the reference, the same way a good exam question accepts a different valid method. So they take the attacker's candidate, drop it into the real repository, and run the project's own hidden tests. If it resolves the bug, the spec just allowed a second correct answer. If it doesn't, the spec was too loose, and it goes back for another round. Run that whole pipeline across all five hundred instances in SWE-bench Verified, and you get what the paper calls SWE-Proof, a proof-checked twin of the entire benchmark. It's verified under three separate proving systems, so no single tool's blind spot decides the outcome. And this is exactly why we do one of these a day, start to finish: one real paper, walked all the way through. Subscribe if you want the next one showing up the same way.

6:21Eric: Okay, so that fifty-eight from the open, that's Claude. What happens to the other frontier model in here?

6:27Paige: The other frontier model goes the same way, only harder. GPT-5.5 starts at eighty-one point two, passing every hidden test. Send those same patches through the adversarial audit, hunting for one input where they and the gold patch disagree, and it lands at thirty-three point four. That's not a quarter of the benchmark coming apart. That's well over half of GPT's passing patches, turning out to diverge from the real fix somewhere.

6:50Eric: Hang on — "diverges from the real fix" isn't the same thing as "the code is broken," is it? If I'm running one of these patches in production tomorrow, what did I actually just learn?

7:01Paige: That's worth being precise about, because it's easy to overstate. "Overturned" means an auditor found one input, out of an infinite domain, where the agent's patch and the human's official fix behave differently. That difference was checked mechanically, not asserted by an AI's opinion. It's proof the benchmark's own tests were too weak to see the difference. It is not proof that a quarter of AI-written code breaks, the moment it ships somewhere. Now, does swapping tests for proofs just mean "add more rules"? They tried a structured plain-English format first, called EARS — templates like, "when this happens, the system shall do that." Hand an agent one of those, and its test pass rate still looks strong on paper. Under a real formal spec, the model loses almost nothing: under a point for Claude, and under half a point for GPT. Under the English version, Claude loses seventeen points it didn't know it had lost, and GPT loses forty-six.

7:54Eric: So the "shall" statements are decoration. The gap between sounding precise and being precise is enormous.

8:00Paige: And this is where it gets stranger than, "formal specs are just better." Hand a model somebody else's correct formal spec, and ask it to write and prove the patch against it, and Claude climbs from eighty-five to ninety-six. GPT goes from eighty-one to ninety-four. Both models get to proving almost for free, once the target is stated with enough precision.

8:20Eric: Wait — if proving is close to free once you're handed the target, what happens when you make the model write that target itself?

8:28Paige: That's the setting that reorganizes the whole paper. It's the same repo, the same issue, the same budget, and the same proving tools. The only change is whether the agent has to state the target itself. The gain from being handed a correct spec, versus writing your own, is eleven to fifteen points. Left on its own, the best a model manages is roughly its baseline. Every other version of that comparison is flat, or worse.

8:52Eric: So proving isn't the wall. Stating the target is the wall.

8:55Paige: That's the reframe. The specs models write for themselves pass an independent audit, somewhere between forty-six and seventy-two percent of the time, depending on the model. And when they fail, it's almost never because the spec contradicts itself, excludes inputs it shouldn't, or misdescribes a function it's leaning on. Each of those failure types sits under four percent. One property fails, over and over: the spec just doesn't cover enough of what the fix actually touches. That single failure trips up forty-two percent of Claude's own specs, and thirty percent of GPT's.

9:27Eric: Give me the failure, then. What does "doesn't cover enough" look like inside a real patch?

9:33Paige: Inside a real patch, it looks like this: there's a Django instance about a date picker. Type an absurdly large year into a form field, and the server crashes, because a bare integer conversion throws an error type the code doesn't catch. The agent finds this correctly in one pass, and widens the exception handler to catch that error too. It writes a spec — succeeds, or falls back to a placeholder string — and the spec verifies on the first try. All fifteen of the widget's tests pass. It reproduces the crash from the bug report, prints the result, sees no crash, and declares victory.

10:06Eric: So it's done. Tests pass, proof passes, and the crash is gone.

10:10Paige: Except the official fix does something the agent's version doesn't. On that failure, it returns one specific string, "zero, dash, zero, dash, zero," a value built to get rejected the moment it's fed back into the form, so the user sees an error instead of a silently wrong date. Two hidden tests turn on that distinction. The agent's version instead echoes the attacker's huge year, back into something the form will try to parse as reasonable.

10:34Eric: So the thing it printed as proof it had won, was the evidence it had lost, sitting right there in its own transcript.

10:42Paige: And no amount of proving could have caught it, because the abstraction the model chose had only two outcomes: succeeds, or fails. The real requirement has three: succeeds, out of range, or too large to represent at all. The distinction the hidden test needed was thrown out, before a single line of proof got written. The authors have a phrase for it: a faithful proof of an insufficient claim.

11:04Eric: I want to push on that eighty-five-to-ninety-six number, because it's the headline of the whole result, and I don't think it earns quite the shine it's getting. Those "correct" specs the model gets handed weren't written blind. During construction, whoever wrote the ground-truth spec had the gold patch, the buggy code, and the full list of tests that had to flip. And the spec's strength got calibrated, by checking whether a wrong implementation could still sneak past it.

11:30Paige: The spec in that setting is built from the gold patch and the full test list, calibrated against the answer key, not written independently of it. So that eleven-to-fifteen-point jump shows what happens, when you're handed something close to a formalized grading rubric. Whether an engineer's spec, written cold from a bug report with none of that visibility, buys anywhere near as much — the paper doesn't establish that, and I don't think it can from this setup.

11:57Eric: Do we even know if this faithfulness gap shows up, outside a five-hundred-instance benchmark someone already solved once, on an issue where nobody's written the fix or the test yet?

12:07Paige: No, we don't know, and I don't think the paper claims to either. Every one of these instances was picked, because a maintainer had already fixed it and already tested it. What happens on a bug nobody's solved yet, is a harder, separate question this paper doesn't touch. There are a couple of numbers worth holding onto past this. Proving the fix, once the spec exists, costs the model almost nothing, about a point, on top of the resolve requirement. And the agents weren't running out of room to think, either. Out of more than fifty thousand attempts, only a handful ever hit their step limit. Whatever's failing here isn't patience. It's the ability to say, precisely, what "fixed" was supposed to mean in the first place. That eighty-five percent Claude opened with is real. It's just measuring something narrower than "correct." It's measuring, "survives the tests this particular project happened to have, lying around." The ceiling on trustworthy AI-written code isn't proving anymore, because proving turned out to be close to free. It's whether anyone — a model, or the person writing the ticket — can state precisely what the code was supposed to do. Two things to take with you from this one. Roughly a third of Claude's test-passing patches, and well over half of GPT's, diverge from the official fix, once an adversarial auditor goes hunting for a single input where the two disagree.

13:25Eric: And handing either model a correct formal spec pushes it to around ninety-five percent — but making it write that spec itself gains nothing. Both models are bad at deciding what counts as the whole behavior, not at proving a claim once it's stated.

13:40Paige: And the one I'd hold loosest — that ninety-five percent depends on a spec built with the answer key already in hand. So how much of that gain survives, when nobody's holding the answer key, is still open.

13:52Eric: If your job is deciding whether to trust one of these agents, on a real ticket tomorrow, that's the actual question underneath all of this. It's not, "can it prove its code correct?" It's, "can anyone, including the model, tell you precisely what correct was supposed to mean," before a single test gets run?

14:11Paige: That's the question, and it's the one this whole paper circles. The full version of this one's up on paperdive.ai, with every term in here — axiom, faithfulness, all of it — tappable for a plain definition, plus links out to the related verification papers if you want to keep pulling the thread.

14:29Eric: Quick housekeeping. The script was written by Anthropic's Claude Sonnet 5, and then refined by OpenAI's GPT-5.6 Sol. Paige and I are both AI voices from Eleven Labs, and we're not affiliated with any of those companies. The paper is "SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?" by George Ma and their colleagues, posted September 18th, 2026. So if a model can prove its code correct, against whatever it wrote down as the requirement — and it just doesn't know what it left out — how much should any of us actually trust the word "resolved," the next time we see it on a leaderboard?