Why an AI Called Fourteen Broken Figures Perfect, And What It Reveals About Test-Time Compute

Also available as a plain-text transcript page.

0:00Juniper: An AI was shown fifteen figures with obvious defects — section titles printed right on top of each other, whole blocks of content running clean off the edge of the page — and it rated fourteen of the fifteen perfect.

0:13Eric: Quick heads up before we start — this is an AI-made explainer, both voices included.

0:18Juniper: And the model wasn't being lazy, and it wasn't biased. It literally could not see the mistakes. By the end of this you'll understand exactly why, and why that same blind spot is probably sitting inside the AI tools writing your code and building your slides right now.

0:35Eric: The paper is called "Interaction Scaling," out this month, and it makes a sharp claim. There's a way to spend computing power to get better answers out of an AI that the whole field has been half-blind to. And the reason it stayed invisible is the exact same reason that judge called broken figures perfect.

0:53Juniper: So let me set up the question, because it starts somewhere familiar. Once you've trained a model, its knowledge is frozen. You can't cheaply make it smarter. But you can spend more effort at the moment you ask it something — what people call test-time compute. Think of a student who's done all their studying. The exam's in front of them. You can't send them back to school, but you can hand them more time and more scratch paper.

1:19Eric: And there are two famous ways to hand over that scratch paper. One, let the model think longer — write out its reasoning before it answers. Two, let it try many times and keep the best attempt. Thinking harder, or buying more lottery tickets from the same machine. And the obvious third move is just as intuitive: let the model look back at its own work and fix what's wrong. That's the whole self-correction idea.

1:45Juniper: And that obvious move is where the paper drops the hammer. Because all three of those — think longer, try more, re-read and revise — pull from the same well. The frozen weights, and the prompt you gave it. Nothing new comes in.

1:58Eric: Wait — but re-reading feels different to me. When I reread my own essay I catch typos I missed the first time. Why doesn't the model?

2:07Juniper: Because you and the model aren't in the same situation. When you reread, you often bring in something the first pass didn't use — you say it out loud, a fact clicks. The model re-reading is just running its own output back through the same weights. There's a result from information theory that pins this down: you can't create new information about something by reprocessing what you already have. Downstream steps can preserve information or lose it. They can't manufacture it. No amount of rearranging the furniture in a room adds a chair that was never there. So before the fix — why can't the model just re-read its draft and repair it?

2:47Eric: Because re-reading is post-processing. It reshuffles what's already inside. It can't add what was never there.

2:54Juniper: And the authors are careful here. That theorem motivates why internal methods plateau. It doesn't prove it. The proof is in their curves. So here's the third axis. Instead of staring at the crossword harder, or guessing more answers, the model submits the puzzle and gets told which squares are wrong. Something outside its own head observes what it actually did — runs the code, renders the page, measures the layout — and reports back. That observation carries information the weights never held. They call it interaction scaling.

3:27Eric: And the setup they test it with is deliberately bare. Three players, no training, no fancy controller. A proposer — a frozen frontier model that produces the artifact. An instrument — something that actually executes or measures it. And a reviewer — the same model, now reading the instrument's report and turning it into a list of concrete defects. Proposer fixes, loop repeats, and you keep the best-scoring version.

3:53Juniper: One detail matters there. Keeping the best-scoring version means a reviewed answer can never score worse than the first draft. So they've closed off the easy objection before anyone can raise it.

4:05Eric: Now the coding test comes into play. Fifteen hard programming tasks, a fixed budget, four strategies racing. Reasoning-only flattens out at seventy-three percent. Best-of-N — try many, keep the best — is the sharp one, because its judge is the actual test suite. It's a perfect judge. And it still caps at eighty-seven percent.

4:25Juniper: Which is strange, isn't it? A flawless judge should get you to the top.

4:30Eric: You'd think. But a judge can only pick from essays the student actually wrote. If the winning idea never occurred to the model across all those attempts, no amount of perfect judging conjures it. The ceiling isn't the judge. It's that selection can't create a candidate that was never drawn.

4:48Juniper: And the interaction loop?

4:50Eric: It climbs straight through both of them to a hundred percent. Every task gets solved. And here's the part that got me — zero run-to-run variance. Single-shot, every model wobbles depending on the random draw. The reviewed version lands on the same perfect score every time, across Claude, Qwen, and GPT-5. Sonnet alone jumps thirty-three points.

5:11Juniper: That's the first real payoff. Internal effort plateaus, and external observation keeps climbing. If you want every major AI paper broken down like this, daily, subscribe — that's the whole reason this channel exists. But "let it check against reality" is only half the story, and it's the easy half. Because you can run a grounded loop and have it make things worse. The question is when. Here's the cleanest experiment in the paper. Hold everything fixed — same tasks, same model, same number of review passes. Change only one thing: what the reviewer is allowed to see.

5:48Eric: On the coding side they tried three reviewers. One re-reads its own code, no tool. One gets a linter, which scans the source. And one gets the actual results of running the program.

5:59Juniper: And the linter is the interesting one, because a linter is genuinely grounded. It's a real measuring instrument. But think of airport security. A metal detector is real, it observes something true — and a ceramic knife walks right through, because metal is all it sees. The bugs they seeded were runtime logic errors. They leave no trace in the source code's form. So the linter, grounded as it is, catches nothing the plain re-read didn't. The execution feedback, which actually watches the program run, points straight at the broken line and converges about two and a half times cheaper.

6:34Eric: So it's not whether the tool is grounded, Juniper. It's whether the tool can observe the thing that's actually broken.

6:42Juniper: That's the coverage principle, and it's the spine of the whole paper. A grounded instrument helps exactly as far as it can see, and no further. Then they run the visual version, and this is where it turns. Same idea — give the reviewer of a slide or a figure one of two instruments. Option one, a tool that measures the real geometry, reading exactly where every element landed on the page. Option two, a vision model looking at a screenshot.

7:09Eric: And the arrow points in opposite directions depending only on which one you pick.

7:13Juniper: They really do point opposite directions. The geometry reviewer sharply cuts layout defects. The screenshot reviewer, on slides, actually increases them — blind to the true positions, its edits break alignment about as often as they fix it. A reviewer that can't see the flaw is worse than no reviewer at all.

7:33Eric: And there's a crack here the authors flag themselves. The geometry tool that drives the fixing is the same tool that scores the result. Same instrument on both ends. Hold onto that. It's the honest weak point, and it comes back.

7:47Juniper: But that same idea — that the instrument decides what's visible — has a second, sharper edge. Because it doesn't just govern the loop that fixes your work. It governs the metric that grades it. And that's where those fourteen perfect figures come back. So picture a machinist cutting a part. While they work, they need a caliper to check the dimension. Fine — that's the fixing side. But at the end, quality control has to certify the part. If the inspector just glances at it from across the room, two things happen. They pass parts that are off by a hair. And worse, they can't tell whether the machinist's careful work made any difference at all. A blind inspector makes real improvements invisible.

8:32Eric: And the standard way the field inspects AI-generated visuals is exactly that glance from across the room. Show a vision model a screenshot, ask "is this good?"

8:42Juniper: Now, what is a screenshot? It's a flat photo, fixed resolution, of the rendered result. So anything that spilled off the edge of the frame is simply gone before the image ever reaches the judge. And anything too small — a slight overlap, a hairline collision — falls below what the model can resolve. The defect isn't misjudged. It's absent. So they ran the probe on fifteen dense academic-paper figures. The screenshot judge, the standard method, rated fourteen of fifteen perfect. Then they pointed a grounded tool at the same figures, one that reads the actual bounding boxes of every element.

9:20Eric: And how many were actually clean?

9:23Juniper: Three. Only three of fifteen came back clean.

9:26Eric: Wait — the judge said fourteen, the tape measure said three?

9:31Juniper: It was fourteen against three. And one of those "perfect" figures — it's an AlphaFold-style architecture diagram — had two section titles printed directly on top of each other. "Processing" superimposed over "Representation." Axis labels colliding with the matrix borders. It was all right there in the image. The judge said perfect, because the errors it could have caught were the ones cropped off the frame, and the ones still in frame were too small to read.

9:59Eric: It's like proofreading a document from a photo that chopped off the bottom half of every page. You'll swear it's flawless. You never saw the errors.

10:07Juniper: And once they scored with a grounded metric, the fix-and-recheck loop removed between forty and about three-quarters of the measured defects across figures, slides, web pages, and animations. One slide from a GAN paper went from content overflowing the frame by more than half its height, six defects, down to zero. And here's the sting, Eric — those are gains the screenshot judge could not detect at all. You could be running a real improvement loop and see nothing move, because your ruler is blind.

10:37Eric: So the one-line version: grounding has to hold on both sides. The tool that fixes your work, and the tool that grades it. Miss the second, and success looks identical to failure. Which brings me to what I don't buy yet. That same grounded metric drives the fixing and scores the result. Same instrument on both ends. So some of that defect reduction is mechanically guaranteed. You're optimizing the exact number you then report as your win.

11:04Juniper: That's fair, Eric, and the authors say it outright — it's the principal caveat they name. They do offer defenses. The defects are real and above the threshold a person would notice, and the ungrounded reviewer moves the same metric the wrong way, which a pure optimization artifact wouldn't. But there's a gap here —

11:23Eric: But there's no human-preference study. Nobody sat people down to confirm that fewer measured defects means a slide a human actually likes better. So what they've shown, cleanly, is that the loop optimizes a proxy. That the proxy tracks real quality is an argument, not a measurement.

11:40Juniper: I'll concede that. They haven't closed it, and they say so.

11:43Eric: And there are two more things worth naming, Juniper. The fourteen-of-fifteen set is a deliberately hard, curated batch. It's an existence proof that the blindness is real, not a measure of how often it bites in normal use. And the perfect coding scores are on fifteen tasks. They're real, but small. A hundred percent on fifteen is a different animal from a hundred percent on fifteen thousand.

12:06Juniper: That's all true. And to their credit, two of the visual categories — video editing and factual research — they report as honest negatives. Those saturate single-shot, the loop adds nothing significant, and they refuse to dress it up. That restraint is why I trust the rest. So let's step back. The reframing is the thing to keep. Test-time compute isn't two moves, think longer and try more. It's three. And the third one, interaction, is different in kind, because it's the only one that imports information the model never had.

12:37Eric: And the practical lesson isn't "add more loops." It's ground the loop you add — make sure your instrument can actually see the flaw that matters, on the fixing side and the scoring side.

12:48Juniper: And now that sentence from the start actually decodes. An AI called fourteen of fifteen broken figures perfect, not because it was fooled, but because the picture it graded had the mistakes cropped off before it ever looked. The failure was in the ruler, not the judgment.

13:04Eric: So here's the one to argue about. When we measure whether these self-improving AI systems are getting better, should we demand a grounded instrument — measure the real thing — before we trust any reported gain? Or is that too strict, and a good-enough model judge is fine for most of it? Tell us where you land.

13:24Juniper: The full annotated version is on paperdive.ai — every technical term tap-to-define, with links to the related papers grouped by theme.

13:32Eric: Quick housekeeping: this script was written by Anthropic's Claude Opus 4.8, Juniper and I are both AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper is "Interaction Scaling," posted July thirteenth, 2026, and we're recording the day after.

13:50Juniper: Turns out "looks perfect" and "is perfect" were never the same measurement.