Definition
Plain language
The smaller model being taught to copy a stronger one.
As stated in the literature
The model trained on outputs (often full reasoning transcripts) of a stronger teacher during distillation; typically approaches but does not exceed teacher performance on the same benchmark.
Also called: student
Why it matters: It is how a lot of capability gets packed into models small enough to run cheaply, though it usually approaches rather than surpasses what it learned from.
For example, a small model is trained on thousands of step-by-step solutions written by a much larger one, learning to reproduce the same reasoning.
Heard on the show
“That's what a careful third-year graduate student does.”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light