distillation

Definition

Training a smaller model to imitate a bigger one, hoping to inherit much of its skill.

A training procedure that transfers behavior from a teacher model to a smaller student by training the student to match the teacher's outputs or intermediate signals.

Also called: distill, distilled, self-distillation, distilling

Mentioned in 9 episodes

1. 078
   Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
2. 071
   When the Model Is Fine and the Plumbing Is Broken: Fixing Agents at the Interface
3. 047
   When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
4. 041
   When the Iteration Teaches the Model to Skip the Iteration
5. 027
   When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure
6. 017
   When the Agent Grades Its Own Homework: A Brutal New Benchmark for AI Workers
7. 013
   Why Search Keeps Rediscovering the Same Workflow, and What That Means
8. 008
   Why Long-Horizon AI Agents Get Stuck, and a Milestone-Based Fix That Helps
9. 002
   An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light

Related concepts