Definition
Plain language
Training a smaller model to imitate a bigger one, hoping to inherit much of its skill.
As stated in the literature
A training procedure that transfers behavior from a teacher model to a smaller student by training the student to match the teacher's outputs or intermediate signals.
Also called: distill, distilled, self-distillation, distilling
Why it matters: It's the main way frontier model capabilities get compressed into smaller, cheaper models that can actually be deployed at scale.
For example, a 7-billion-parameter student model is trained to match the next-token probabilities of a 70-billion-parameter teacher on millions of prompts.
Heard on the show
“The next agent inherits that distilled record and a relevant slice of background knowledge, not the raw transcript.”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light