Glossary · Term

distillation

← all terms

Definition

Plain language

Training a smaller model to imitate a bigger one, hoping to inherit much of its skill.

As stated in the literature

A training procedure that transfers behavior from a teacher model to a smaller student by training the student to match the teacher's outputs or intermediate signals.

Also called: distill, distilled, self-distillation, distilling

Why it matters: It's the main way frontier model capabilities get compressed into smaller, cheaper models that can actually be deployed at scale.

For example, a 7-billion-parameter student model is trained to match the next-token probabilities of a 70-billion-parameter teacher on millions of prompts.

Heard on the show

“… One more line for speed: a distillation step teaches the model to denoise in one or two strides instead of dozens, like a sculptor taking …”
Episode 206 — How Four-Second Clips Become Hours of Playable AI Soccer

Mentioned in 35 episodes

  1. 206
    How Four-Second Clips Become Hours of Playable AI Soccer
  2. 200
    The One Mechanism That Turns Twenty AI Clones Into an Actual Team
  3. 194
    How a Robot Builds a Debugging Notebook It Can Read, Edit, and Hand to Another Robot
  4. 193
    Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer
  5. 192
    A 32B Open Model Matched Frontier Systems By Learning to Take Notes
  6. 189
    Why Phone Agents Ace the Test and Crash on Your Actual Phone
  7. 186
    How a Frozen Model Went From 2% to 77% on Physics Puzzles — Without Retraining
  8. 183
    Why You Can't Fine-Tune Foresight Into an AI Agent
  9. 168
    When Turning Experience Into Code Makes Your AI Agent Dumber
  10. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  11. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  12. 156
    Why More Human Demonstrations Made a Computer-Use Agent Worse
  13. 155
    Why a Flawless Demo Makes a Worse Computer-Using Agent, And the Fix
  14. 142
    Training a Tiny Model to Run the Plumbing Between an Agent and the World
  15. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  16. 131
    Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
  17. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
  18. 126
    How Coding Agents Can Mine Their Own Failures Into a Self-Targeting Curriculum
  19. 121
    When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model
  20. 114
    Agents That Rewrite Their Own Weights Instead of Just Taking Notes
  21. 111
    How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
  22. 110
    How an Agent Got 44 Points Better by Mining Its Own Scratch Paper
  23. 106
    Giving Agents a Notebook Instead of New Weights: How ExpGraph Lets Frozen Models Learn
  24. 099
    How an Open-Book Trick Teaches a Model to Catch Its Own Mistakes
  25. 082
    Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick
  26. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
  27. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  28. 071
    When the Model Is Fine and the Plumbing Is Broken: Fixing Agents at the Interface
  29. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  30. 041
    When the Iteration Teaches the Model to Skip the Iteration
  31. 027
    When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure
  32. 017
    When the Agent Grades Its Own Homework: A Brutal New Benchmark for AI Workers
  33. 013
    Why Search Keeps Rediscovering the Same Workflow, and What That Means
  34. 008
    Why Long-Horizon AI Agents Get Stuck, and a Milestone-Based Fix That Helps
  35. 002
    An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light

Related concepts

AI Efficiency & Cost Knowledge Distillation