Making a Vision Model Better by Showing It Blurry Images

Also available as a plain-text transcript page.

0:00Bella: Here's a claim that shouldn't survive contact with reality. You can make a vision-language model better at spotting tiny details in big images by never letting it see the details. Show it blurry, downscaled, grainy copies of every image while it trains, with no labels, no answer key, no reward, and no bigger model to imitate. And a four-billion-parameter model climbs from about seventy-one percent to about seventy-seven and a half on six benchmarks built specifically to test fine visual detail. Seventy-seven and a half puts it above an open model roughly sixty times its size, and above GPT-5.4.

0:35Finn: And the blur isn't a side effect of something else. The blur is the whole supervision signal. Take it away, train the same way on clean images, and the identical procedure makes the model worse than leaving it alone.

0:48Bella: So by the end of this, you'll know where that free supervision actually comes from, why the obvious version of the idea collapses, and the one result in here that changes what "more training signal" even means. And this matters because of a question sitting under a lot of recent post-training work. Once a model is already good, where does the next unit of supervision come from? Human labels get expensive fast, automatic verifiers only exist for things like math and code, and renting a frontier model as a teacher costs real money.

1:20Finn: Okay. So the machinery everyone's using here is called on-policy distillation, and Bella, this is the part where the details matter, because the whole paper is a hack on one hidden assumption inside it.

1:32Bella: Right. So think of a tutor marking up your own essay, rather than handing you somebody else's finished essay to copy. The model writes an answer itself, token by token, and at every single position, the teacher says what it would have predicted there. And a language model doesn't just pick a word, it produces a spread of probabilities over all the words it might pick. So the teacher's correction is rich. It's not "the answer is red." It's "at this word you put forty percent on red and thirty on orange, and I'd have put eighty-five on red." One correction per token. A two-hundred-token answer gives you two hundred corrections, at exactly the states the model actually visits when it's working on its own.

2:12Finn: Mm-hm. Which is why it's so much denser than reinforcement learning, where you get one scalar at the end saying you won or you lost.

2:20Bella: Exactly. But there's a load-bearing assumption in there, and the paper names it: informative teacher-student asymmetry. For the tutor's opinion to be worth anything, the tutor has to know something the student doesn't. Classically you buy that the expensive way, with a bigger, stronger teacher model. Then a wave of recent work found a cheaper route — use the same model as its own teacher, but slip the teacher something privileged: the reference answer, a verified reasoning trace, or ground-truth boxes marking the region of the image where the answer lives.

2:52Finn: And here's where I'd have gone wrong, and I think a lot of people would. My instinct is, forget the privileged extras, because a model is stochastic, and its own averaged weights aren't quite the same as its current weights. So just make the model its own teacher, same image, same question, and let it bootstrap. Free supervision, no annotations, done.

3:13Bella: It doesn't work. They ran exactly that, and it's the cleanest negative result in the paper — teacher and student both looking at the same clean image, everything else identical. The base model in that protocol scores about seventy and a half. After training that way it scores sixty-five point two.

3:30Finn: So training it actively broke it.

3:33Bella: It went backwards, yes. And the authors' explanation is one line: with no information gap, the objective amplifies the teacher's confident errors instead of correcting the student's. It's a photocopy of a photocopy. There's no new information entering the loop, so whatever the model already believes just gets sharper — including the parts it's confidently wrong about.

3:54Finn: So the field's read on that was, fine, you need real asymmetry, and asymmetry is a resource. You go acquire it — a bigger model, a human annotator, a verifier.

4:04Bella: And that's the inversion. Their thesis line is one sentence: rather than adding privileged information to the teacher, we subtract information from the student. Asymmetry isn't a resource you buy. It's a relative difference between two sides, and you can open the gap by moving either side. Adding to the teacher costs money. Withholding from the student is free.

4:24Finn: Huh. Okay, so concretely. Same model — the teacher looks at the sharp, full-resolution photo, and the student answers the question while looking at a shrunken, noisy copy of that same photo. And the disagreement between them is the lesson.

4:39Bella: That's it. The image on screen is the whole method — one photograph split down the middle, razor-sharp on the left, downscaled and grainy on the right. The coach watches the drill in full daylight, while the player runs it wearing smudged glasses, but the coach isn't better at the sport — the coach just has better eyes on the same play, and the difference between what the coach sees and what the player sees is the entire curriculum.

5:04Finn: So what's the recipe, in practice? Because "degrade the image" covers a lot of ground.

5:10Bella: It's almost disappointingly mundane. Downscale the student's image to somewhere between point-three and point-six times the original resolution, and about half the time add a little Gaussian noise — roughly a ten percent pixel jitter, so visible grain, not destruction. That's the winner out of sixteen-plus operators they tested.

5:30Finn: And that downscale isn't just softness, right? Because of how these models take images in.

5:36Bella: Right, that's the part that makes it more than cosmetic. A vision-language model doesn't see a picture. It chops the image into patches and turns them into visual tokens, and the number of tokens scales with resolution. They downscale and never resize back up, so the student is literally handed fewer visual tokens — less raw evidence in the input sequence. Which also means — and this is a nice grace note — the student's forward passes and rollouts are cheaper than the baseline they're compared against. The method improves accuracy and reduces compute per step.

6:09Finn: Okay, and let's land the headline properly, with the scoping, because this is where these stories usually get oversold.

6:17Bella: Agreed, so let's be precise. They trained a four-billion-parameter Qwen3.5 for a hundred and thirty optimizer steps — one epoch over a twelve-thousand-sample subset of FineVision, image-question pairs with no answers attached. Average across the six fine-grained perception benchmarks goes from seventy point seven to seventy-seven point four. That seventy-seven point four beats a two-hundred-thirty-five-billion-parameter open model at seventy-five point eight. It also beats GPT-5.4 at seventy-two point eight. And it lands within a rounding error of a three-hundred-ninety-seven-billion model, which is a coincidence, but a pleasing one.

6:55Finn: And what it is not, is a four-billion model that's smarter than GPT-5. It's a four-billion model that got specifically sharpened at finding small things in large images, measured on benchmarks about finding small things in large images. Gemini-3.5-Flash and Gemini-3.1-Pro still beat it, by under two points, but they beat it.

7:14Bella: That's the honest frame, and saying it up front is what buys us permission to enjoy the number. And if you want every major AI paper broken down like this — mechanism first, caveat included — subscribe, we do one a day.

7:27Finn: So the mechanism is simple enough that the interesting question becomes an empirical one. How much should you handicap the student? And that's the part of the paper I want to sit in, because it pays off in a single measurable quantity that has a sweet spot — and a trap that I did not see coming.

7:44Bella: Before that, though, there's one design choice that isn't a footnote: how you measure the disagreement.

7:51Finn: Yeah, and normally I'd skip a divergence ablation. Here it's actually tied to the mechanism.

7:57Bella: It is. So you need a number for "how far apart are these two probability spreads." The default is KL divergence, and the non-obvious fact is that KL has a direction. Run it one way — the coverage-seeking direction — and the student gets punished for putting near-zero probability on anything the teacher thinks is plausible. It's forced to take every one of the teacher's beliefs seriously.

8:19Finn: Including beliefs about details that aren't in the student's image at all.

8:24Bella: Which is pressure to bluff. The teacher's confidence is partly built on pixels the student was never given. So that direction tells the student: fake it. Run it the other way — mode-seeking — and the student is allowed to pile everything onto whatever it already believed, and quietly discard the rest of the teacher's opinion. That throws away the signal you built the whole method to create.

8:47Finn: So they take the middle.

8:49Bella: Jensen-Shannon divergence, the symmetric compromise, and it wins on average and on every reported benchmark — seventy-six point oh five, versus seventy-five point five for one direction and seventy-four point seven for the other. It also has a plain practical virtue: it can't blow up when the two spreads barely overlap, which matters a lot when you've engineered them to disagree on purpose.

9:13Finn: Okay, so quick checkpoint on where we are. Distillation needs a knowledge gap. Everyone bought the gap. This paper manufactures it by degrading the student's view, and measures it with a divergence that doesn't demand the student hallucinate detail.

9:29Bella: And so before the results — where does the supervision actually come from, in one line?

9:34Finn: From the difference between two views of the same image. Nothing external enters the loop.

9:40Bella: Which sets up the real study. This paper's second contribution is that it treats the augmentation itself as the object of research. Sixteen-plus operators, sorted into four families. First, information reduction, like blur and downscaling and dropping visual tokens. Second, geometric, like rotation and translation and cropping. Third, photometric, like brightness and saturation and hue. And fourth, occlusion, like erasing random patches. And then full matched training runs whose only difference is the distortion recipe — same data, same optimizer, same step budget, same eval.

10:16Finn: And the first finding is broad. All four families beat the base model, and all four beat the collapsed same-image version.

10:24Bella: All four. Information reduction is best, at about seventy-five and a half. Occlusion is weakest, at about seventy-two and a half. That's against a base of seventy point six, and that collapsed sixty-five point two. And the one I keep coming back to — brightness and saturation jitter works. It changes no geometry, deletes no content, removes nothing you'd call evidence, and it still produces useful asymmetry.

10:49Finn: Which honestly makes me suspicious of the collapse explanation. If merely nudging the colors is enough to make the objective productive, then "amplifying confident errors" might not be the real story about why the same-image version degrades. Minimizing divergence between a model and a smoothed copy of itself on identical inputs has a trivial degenerate solution regardless of information content. That could be an optimization pathology rather than an epistemic one, and the paper spends one sentence on it.

11:20Bella: That's fair, and they don't distinguish those two stories. What they do rule out is the other obvious explanation, and they rule it out against their own interest. The teacher here is an exponential moving average of the student — a slowly updating running average of its own recent weights, borrowed from BYOL and DINO. A lot of papers would have sold that momentum teacher as the source of the magic. They froze the teacher entirely at the base model, so it never improves at all, and it still recovered ninety-three percent of the gain. Seventy-five point nine five versus seventy-six point three five.

11:56Finn: So the moving-average teacher is a stability convenience, and the asymmetry is the substance. Good. Now the trap.

12:03Bella: Right, the asymmetry is doing the work and the running average is just scaffolding. So what's the trap?

12:09Finn: So how do you compare a blur radius to a crop scale to a saturation shift? The units aren't comparable. Their move is to stop plotting nominal strength and instead measure the actual thing — the token-level disagreement between teacher and student over the first ten training steps, one number that captures the real size of the handicap. And now blur, downscaling, and cropping all live on the same axis, and you get a clean inverted U. Accuracy rises as the gap grows, peaks around a specific measurable point, then falls. Downscaling peaks at the point-three-to-point-six band, blur peaks in the middle of its range, and visual-token dropping shows the same shape.

12:47Bella: So too small a gap gives you no signal, and too large a gap gives you... what, exactly?

12:52Finn: That's the question, and cropping answers it. Cropping produces the largest teacher-student gap in the entire study — the biggest handicap they measured anywhere.

13:03Bella: Okay, and the biggest handicap in the study should be either the best result or a total disaster. Which one is it?

13:09Finn: It performs the worst, at sixty-seven point four four percent, barely two points above the collapsed baseline that made the model worse. And it falls monotonically as you crop harder — seventy-one and a half, sixty-eight point eight, sixty-seven point four.

13:25Bella: So the biggest disagreement in the whole study produced almost the worst model.

13:30Finn: And here's the part that kills "bigger gap, better gap" outright. The mildest crop has a measured gap that coincides with the best blur and downscale settings — same gap size — and it still trails them by more than three points. Same magnitude. Wildly different value.

13:46Bella: So the magnitude of disagreement is not a proxy for the value of disagreement. Which is where the coach analogy earns its keep. With smudged glasses, the player half-sees the play, reaches for the right move, misses by a bit, and the correction lands on something they were actually attempting. Now blindfold the player and take the ball off the field. The coach still has a confident opinion about the right move — but the player wasn't struggling, the player was guessing, and there's nothing to learn from the gap between an informed answer and a coin flip.

14:18Finn: Because cropping can delete the exact region of the image the question is about. The student's input becomes unanswerable. And you can see it in which benchmarks break: going from moderate to strong cropping costs about two points each on the ones that need localized evidence.

14:34Bella: Which gives the paper its operating window, stated as a constraint rather than a boast. The degradation has to remove enough task-relevant evidence to create a signal, but not so much that it changes the underlying question or makes recovery impossible.

14:48Finn: And there's a second thing this predicts that I like, because it's a different kind of evidence. The privileged-information methods that teach a model where to look — feeding the teacher ground-truth regions — improve perception and damage reasoning. One of them, at four billion parameters, drops a visual math benchmark by twenty-seven points.

15:08Bella: Twenty-seven. And this method improves math reasoning by about three and a half points at the same time as it improves perception by nearly six. Which tells you what the signal actually teaches. It isn't teaching the model where to look; it's teaching it to see harder — to reconstruct, from thin evidence, the answer that rich evidence supports. That's a general perceptual pressure, so it doesn't fight the rest of the model. It also holds at nine billion parameters, with somewhat smaller gains.

15:37Finn: Which brings me to the thing that's been bothering me since the method slide, Bella, and the paper never addresses it.

15:44Bella: Then let's put it on the table, because an unexamined mechanism is worse than a wrong one.

15:50Finn: The whole virtue of on-policy distillation is that supervision arrives at the states the model actually visits. No train-test mismatch. That's the pitch. And this method preserves that perfectly on the text side — the teacher grades the student's own generated tokens. But on the image side, they break it deliberately and completely. The student trains on blurry inputs it will never receive again. At evaluation it gets clean, full-resolution images. So the model is being pulled toward a distribution derived from an input it never actually sees, and it's tested in conditions it never trained in.

16:22Bella: The folk story would be, you practice with a weighted bat and you swing faster in the game.

16:27Finn: Right, and that's exactly the kind of reasoning that's often wrong, and the paper doesn't investigate it at all. It works. Nobody in the paper explains why it should. And that connects to my other reservation, which is about the augmentation map itself. Their three findings are: asymmetry helps, moderate is best, and the gap has to stay task-consistent. But "task-consistent" is diagnosed entirely after the fact — it's invoked to explain the one family that broke the trend. There's no independent measure of it. Hand me a brand-new distortion, and I can't tell you in advance whether its gap will be informative or garbage. That's a compelling narrative, not a criterion.

17:04Bella: I'll concede both. The mechanism story for why blurry training transfers to clean testing is unexamined, and the task-consistency rule is post hoc. I'd add two smaller things in the same spirit. The gap measurement is a single scalar taken from the first ten training steps, on analysis runs whose total step budget the paper never states, and it's used as the x-axis of the central figure. And the design-space sweep runs at a cheaper evaluation setting than the headline table. And all of it is one model family, at two scales, on training data drawn from the same natural-image domain the benchmarks probe.

17:39Finn: Which is the honest version of the leaderboard line. A tuned specialist on in-domain data beating untuned generalists on a narrow slice.

17:47Bella: And they should get credit for how much of that they volunteered — the frozen teacher, and the geometric caveat flagged before they tested it. They also reported the best of thirteen checkpoints for the unstable baselines, while reporting their own final checkpoint, and they labeled that generous.

18:05Finn: Also worth being straight about the lineage. Weak-augmentation predictions supervising a strongly-augmented input is Mean Teacher and FixMatch. That structure is nearly a decade of semi-supervised vision. What's new is porting it into a token-level generative objective, where nobody wants invariance to blur — they want recovery of the answer the sharp image supports — and then actually mapping which distortions work there.

18:30Bella: So back to that opening claim, which you can now fully decode. A four-billion model trained only on blurry copies of images beat a model sixty times its size on visual detail, because the difference between two views of the same photograph is supervision, and nobody had to pay for it. And the bigger idea is the reframe: a teacher's advantage isn't a resource you acquire, it's a difference between two sides — and you can widen it from either end, as long as you don't widen it so far that the question stops being answerable.

19:01Finn: So which is it — a general recipe for label-free supervision that'll show up in every post-training stack next year, or a perception-specific effect that works because these benchmarks reward squinting? If you've ever tried to fine-tune a vision model on a pile of unlabeled images, you already have an instinct about that, and I'd like to hear it in the comments.

19:22Bella: The full annotated version of this episode is on paperdive.ai — every technical term tap-to-define, with links to the related papers grouped by theme, and the weekly roundups.

19:33Finn: Quick housekeeping: the script was written by Anthropic's Claude Opus 5, Bella and I are AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "Self-Supervised Visual On-Policy Distillation," by Yijiang Li and their colleagues, posted August 14th, 2026.

19:50Bella: And the result I'm watching for is someone running this recipe on a vision encoder that isn't Qwen's. If that point-three-to-point-six sweet spot shows up again there, they didn't find a trick — they found a map.