Coding Models Can Find the Bad Line, They Just Won't Delete It

Also available as a plain-text transcript page.

0:00Hope: A developer fixed a bug by replacing a single line of code. A frontier model fixed the same bug by keeping that line, tucking it into an else branch, and building a new path above it. Both patches passed exactly the same tests. Both got recorded as resolved. And that's not a one-off. It's a measured, named, quantified habit across every leading coding model on the SWE-bench leaderboard. By the end of this you'll know why the diffs your coding agent hands you keep getting bigger, and where inside the model the failure actually lives. Because when these researchers wrote tests that fail if the old code is still sitting there, frontier success rates fell from about 63 percent to about 42.

0:44Tyler: Right, and the reason that number matters beyond the benchmark is simple. Everybody quotes resolution rate on SWE-bench Verified as the score for "can this thing do real software engineering." But maintainers keep bouncing the pull requests. There's a METR study the paper leans on where they took 296 agent pull requests that had already passed the SWE-bench tests, and the actual merge rate came in 24 percentage points below the benchmark score. So there's a gap, and this paper goes looking for a specific mechanical behavior inside it.

1:20Hope: Mm-hm.

1:21Tyler: And, umm, my first guess was the boring one. Localization. Finding the right code in a repo of a thousand files is famously the hard part of the job. The issue text almost never names a file. So if a model is failing to remove something, the obvious story is that it never found the thing in the first place.

1:41Hope: That's the story the paper kills in about one table. They took every deletion the human developer made and asked three nested questions. First, did the model edit the right file? Second, did it edit somewhere inside the right function or class? And third, did it remove the exact line? Because each level sits inside the one above it, the drop between levels tells you where the capability breaks. It got the right file over 92 percent of the time. It got the right enclosing scope roughly 70 percent of the time. It got the right line under 52 percent of the time. So it walks into the correct room, it stands in front of the correct wall, and then it doesn't take the wall down.

2:27Tyler: So it's not searching. It's deciding.

2:30Hope: It's deciding. The authors call the behavior deletion avoidance, and they're careful about it. It's an observable property of a patch, not a claim about what the model wanted. But the shape is consistent enough that they can put a name and a number on something people have only been complaining about as a vibe. Which is the whole value here, honestly. "AI code is bloated" is a feeling. "Twenty-nine percent of passing patches wrap the target in a guard, and those patches run about two-thirds larger than the developer's" is a measurement you can track and fix.

3:08Tyler: And that's the thing we do here, one important AI paper every day, start to finish, so subscribe if you want them to keep showing up.

3:18Hope: So what do they write instead? There's a dominant shape, and the authors give it a great name: Guard-and-Go. You keep the old logic exactly where it is, and you add a condition in front of it that catches the case from the bug report. That accounts for 29 percent of all the passing patches they classified. Think of a contractor you hired to take out a wall that's blocking a doorway. They leave the wall standing and frame a new hallway around it. The inspector checks that you can walk from the kitchen to the living room, and you can, so they sign off.

3:54Tyler: Okay, but there's a version of that where the leftover doesn't matter, right? If the new hallway is the only way anyone ever walks, the old wall is just... furniture. Ugly, ignorable.

4:07Hope: And the paper preserved exactly that distinction, which I think is the sharpest teaching moment in the whole taxonomy: two outcomes that look identical in a diff. In one, the guard catches every input that would have reached the old code, so the old code is unreachable. That's dead code, annoying, ignorable. In the other, the guard catches only the specific case from the bug report, and the developer's deleted logic remains the default path for everything else. That one they call Retained Path as Live Fallback, and it's the single largest category, 40 percent of the typed cases, more than the next three combined.

4:48Tyler: So the pothole's still open to traffic. There's just a sign telling trucks to use Elm Street.

4:54Hope: That's it. And the cost isn't the extra lines. It's the extra combinations. Every future reader and every future change has to reason about both routes. There's a second flavor worth naming for flavor alone, which they call Exception Capture Bypass. The old code throws an error, so you catch the error, return a default, and never touch the cause. The symptom leaves the logs. The bug stays.

5:20Tyler: Which is a prescribing cascade with extra steps.

5:24Hope: Pretty much. Now, the honest objection to all of this, and I want to put it on the table before the results rather than after, is that a retained line isn't automatically a bug. A guard can be a completely legitimate alternative repair that happens to solve the issue a different way than the human did. The paper says that outright. Hold onto it, because it comes back and it's sharper than it sounds.

5:51Tyler: So how do you test any of this? Because that's the wall, isn't it. Every existing test the model passed, and the benchmark already said yes, so you can't just assert the patches are wrong.

6:04Hope: So they change the grader instead of the model. And the design choice here is the one I'd steal if I maintained a coding benchmark. They wrote checks that are purely source-level. The check reads the target file off the checkout and fails if the developer's removed code is still present in its enclosing scope. It never imports anything, never executes project code, so framework settings and database fixtures can't muddy it. And they validated each one the way you'd want: it has to fail on the original buggy commit and pass on the developer's real fix. Thirty-four tasks survived that validation.

6:43Tyler: Mm-hm.

6:43Hope: Same tasks, same models, stricter check: of the attempts that passed the original test suites, one third still contained the code the developer removed. And across four current frontier models, closed and open weight, the success rate fell from about 63 percent to about 42. Every single model dropped, between 17 and 24 points.

7:06Tyler: Twenty-one points on average, and nothing about the model changed. You just asked a question the test suite had never been asked.

7:15Hope: That's the part that should sit with anyone building evaluations. A test can assert that behavior is present. There's essentially no natural way in a normal test suite to assert that code is absent. Nobody hid this. It's a blind spot in the shape of the instrument.

7:33Tyler: Okay. So here's where I'd push, if I were the skeptic in the room, and it's the objection they're about to spend the rest of the paper answering. Those 34 tasks are deletion-heavy by construction, and the patches are mixtures of adds and deletes, so maybe deletion just correlates with hard. What you'd really want is a test where deleting is the entire job, and everything else has been stripped away.

8:00Hope: Which is the benchmark. It's called CanItDelete, 200 tasks, and this is where the argument gets airtight. They mined about 79 thousand real file edits that delete lines and add none, so no replacement code is needed. They hand over the full file before the edit, so there's no cross-file search. And they score it with a deterministic evaluator, no model judge, that is occurrence-aware. If a line appears three times in the file and you delete the wrong copy, you get nothing. And commenting the code out doesn't count as deleting it.

8:35Tyler: Good. That last one is exactly the loophole I'd expect a model to find.

8:41Hope: The results, with all the addition work removed and the file handed over: the best model lands at 79 percent. The strongest open-weight models cluster in the mid sixties. The weakest scores 18. And for ten of the twelve models tested, the majority failure mode is incomplete deletion — not deleting the wrong thing, not breaking the file, just... leaving part of it there.

9:05Tyler: One thing I want to flag on the benchmark, because the number travels without it. They deliberately picked the two hundred hardest deletion jobs they could find. Every task spans at least three separate deletion sites, and the median file is about fifteen hundred lines with 34 lines to remove. So "the best model fails one in five" is a stress-test number, not what you'd see on an average Tuesday.

9:31Hope: Agreed, and they say so.

9:34Tyler: Alright. So this is where the paper gets to the thing I keep thinking about, and it's a diagnostic ladder: three rungs, each one removing an excuse. The first rung is telling the model explicitly, not "fix this file." The literal instruction is remove this completely, no guards, no comments, no fallbacks, no replacement code. It moves success between minus two and a half points and plus two and a half. It does nothing.

10:00Hope: Nothing. Which rules out the intent story completely.

10:03Tyler: The second rung is pointing at the region. Tell it which functions matter, and it moves zero to seven points, barely anything. The third rung is handing over the exact spans, the exact lines to remove. Every model moves, some by more than thirty points. And for four of the five, incomplete deletion nearly disappears.

10:22Hope: So put it in one sentence, Tyler, because I think this is the reframe the paper exists for.

10:29Tyler: What models lack is control, not capability. They can find it, they understand you want it gone, and what they don't have is a confident sense of where the deletion ends: which lines belong to the thing you're removing, and which ones are collateral. It's a boundary problem.

10:46Hope: Mm-hm.

10:46Tyler: And then the sting: once you suppress the under-deleting, the over-deleting walks in the door. Think about pulling a weed. There are two separate skills there. One is getting the whole root instead of snapping off the leaves so it grows back. The other is stopping at the boundary and not yanking the tomato plant whose roots are tangled up with it. A single pass rate conflates them, and a gardener who gets aggressive about the first problem starts creating the second.

11:14Hope: And that's what the numbers show.

11:17Tyler: Exactly what the numbers show. Give the best OpenAI-line model the exact lines to delete, and its rate of "removed the target but broke something else" doesn't budge, sixteen percent to sixteen and a half, which caps it at about 80. Give one of the big open-weight models the exact lines, and its over-editing rate actually rises, from about twenty percent to twenty-six, as its retention falls. Along the GPT family in the plain condition, incomplete deletions fall from 114 to 20 while invalid edits after complete removal climb from 14 to 32. Getting better at deletion partly means trading one failure for another.

11:59Hope: So before the optimistic ending, let's lock the spine down. Why isn't this a search problem?

12:05Tyler: Because the models edit the right file over 92 percent of the time and cut the right line under 52.

12:12Hope: And why isn't it an intent problem?

12:15Tyler: Because telling them explicitly, no guards, no fallbacks, moves the score by roughly nothing.

12:22Hope: Which leaves the boundary. And that suggests something hopeful, because a missing skill that specific might just be a missing slice of training data. So the last thing they do is a pilot. They take a 7-billion-parameter model, train it twice under an identical recipe, and the only difference is that one mixture includes about thirteen thousand deletion examples. That's 112 million tokens against a mixture of roughly 16 billion — under one percent.

12:53Tyler: And what falls out?

12:54Hope: If the boundary story is right, you'd expect two things. First, better deletion, obviously, since that's what you trained on. But second, also transfer, improvement on benchmarks the new data never targeted, because deletion shows up inside ordinary bug fixing. And both show up. On the deletion benchmark, success roughly doubles and incomplete deletion drops almost 14 points. And on SWE-bench Verified, which the deletion data was not aimed at, it goes from roughly 25 to roughly 31 — five points, from under one percent of the training tokens.

13:31Tyler: Which is cheap, as fixes in this field go.

13:35Hope: Cheap. But look at where that 14-point drop in incomplete deletion actually went, because the authors don't hide it. About seven points became clean, compliant edits. The other seven became complete-but-invalid edits, with over-deletion up more than six points on its own. Their line is that reducing the additive habit did not remove the disposition to act on the code, and it now surfaces as over-deletion. The model learned to finish the job. It did not learn where to stop.

14:06Tyler: So I want to name the objection that survives all of this, because it's real and it constrains the biggest number in the video. That deletion-sensitive check, the one that dropped frontier scores 21 points? Every one of those checks is derived from the human developer's removal. It asserts that this specific source text is absent from this specific scope. So a patch that fixes the issue correctly by a different structure fails that check by construction.

14:37Hope: That's fair.

14:38Tyler: Which means the 21-point drop measures conformance to the developer's solution, not correctness. The paper knows this, and says it plainly, but the abstract's phrasing reads more like a correctness result than a conformance one. And the thing I'd want and don't have is: of those newly failing attempts, how many would a human reviewer have actually rejected? Nobody counted. The maintainability harm is the same shape. This whole paper is premised on Guard-and-Go patches making codebases worse, and nothing here measures review time, defect rates, or downstream bugs. The harm is imported from cited work and proxied by patch size, and the authors concede that patch size alone doesn't show the added code was unnecessary.

15:25Hope: I'll concede both of those. The strongest version of this paper is not "AI patches are broken." It's "there is a systematic editing habit, it is invisible to the current grader, and here is a cheap way to see it." That's what the evidence supports and no more.

15:42Tyler: And I'd add one on the pilot, since we just called it the optimistic note. There's no data-volume control. They added 112 million tokens of deletion data and gained five points, and there's no arm that adds 112 million tokens of ordinary code data. So some of that transfer could be more data, or more diverse agent trajectories, or distillation from the strong teacher that generated the examples. One 7B model, one run per arm, no variance reported, from a base score of 25 where gains come easier — so the authors call it a signal rather than a solution, and I think that's the right label.

16:21Hope: Which brings me back to that first patch. The developer replaced one line. The model kept the line, dropped it into an else branch, and built a new route above it. And now you can decode all of it: the model found that line, understood it was in the way, couldn't tell where the removal ended, and did the thing that's always safe when you're unsure of a boundary. It added. The bigger claim is that the additive bias here isn't a machine quirk at all. People systematically overlook subtractive changes. English text leans toward addition, and models learned from that text. Then we graded them with a tool that can only check what's present. We built something that shares our worst instinct about code, and then built a grader that can't see the instinct.

17:09Tyler: So where should the fix go? Do we patch the graders, and every coding benchmark from here on ships absence checks alongside its tests? Or is the grader a lost cause and the real fix is upstream, in the training mixture, the way that pilot suggests? Those pull in different directions, and I don't think you get both cheaply. If you've been reviewing agent pull requests all year, you already know which one would help you Monday morning, so say which.

17:37Hope: The full annotated version of this episode is on paperdive.ai, with every technical term tap-to-define and links to the related papers grouped by theme.

17:46Tyler: Quick housekeeping: the script was written by Anthropic's Claude Opus 5, Hope and I are AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "To Add Is Machine, To Delete Is Human," by Amir M. Ebrahimi and their colleagues, posted July 30th, 2026.

18:04Hope: So watch for the first coding leaderboard that publishes a second column for what got taken out. Until one does, the score is only telling you what got added.