OOD

Definition

Plain language

Cases that look different from anything the model saw during training.

Also called: out-of-distribution

Why it matters: Real deployments constantly hit inputs unlike the training data, so OOD robustness is often what separates demo-quality from production-quality models.

For example, a model trained on English news articles is OOD when asked to summarize a 17th-century legal document.

Heard on the show

“The ground truth is real execution, and the queries are held out from training, so it's out-of-distribution by construction.”

Mentioned in 15 episodes

1. 167
   How Teaching an AI to Predict, Not Act, Made It a Better Actor
2. 160
   Training an AI to Take Its Own Notes, So Its Future Self Works Better
3. 140
   When a Reasoning Model Says "Let Me Double-Check" After It's Already Decided
4. 127
   What Diffusion Language Models Were Missing: A Map, Not an Algorithm
5. 111
   How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
6. 088
   Two Levers for Self-Improving AI: When Rewriting Code Isn't Enough
7. 079
   An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models
8. 066
   Why Giving an AI Agent More Tools Can Make It Worse at Using a Computer
9. 051
   Why Parallel Sampling Plateaus, And What Evidence Graphs Do Instead
10. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
11. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
12. 032
    A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking
13. 022
    Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap
14. 017
    When the Agent Grades Its Own Homework: A Brutal New Benchmark for AI Workers
15. 013
    Why Search Keeps Rediscovering the Same Workflow, and What That Means