pretraining

Definition

Plain language

The initial expensive training phase where a model learns from a huge pile of text.

Also called: pretrained, pre-training, pretrain

Why it matters: Almost everything a model can do downstream is shaped by what happened during this expensive initial stage.

For example, the model spends months predicting the next token across trillions of words of books, code, and web pages.

Heard on the show

“They built it on top of DINOv3, a big frozen vision model pretrained on enormous amounts of ordinary images, which already understands objects, motion, and spatial layout.”

Mentioned in 41 episodes

1. 206
   How Four-Second Clips Become Hours of Playable AI Soccer
2. 193
   Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer
3. 183
   Why You Can't Fine-Tune Foresight Into an AI Agent
4. 172
   One Bad Token Can Sink a Model's Math, And You Can Delete It
5. 169
   Why Better Bug Reports Can Make AI Coding Agents Worse
6. 167
   How Teaching an AI to Predict, Not Act, Made It a Better Actor
7. 163
   Why Training Only on Perfect Solutions Cripples a Model's Reasoning
8. 145
   Building Forgetting Into a Language Model With One Extra Line of Code
9. 133
   How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
10. 128
    How a Model Can Earn Full Reward and Still Resist Training
11. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
12. 114
    Agents That Rewrite Their Own Weights Instead of Just Taking Notes
13. 112
    When an AI Agent Cheats Without Being Told: Inside the Meta-Agent Challenge
14. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
15. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
16. 092
    When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
17. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
18. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
19. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
20. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
21. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
22. 076
    Same Model, Organized Differently: How an Agent Architecture Beat Frontier Systems at Research Math
23. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
24. 070
    When Models Know the Answer But Say the Wrong Thing Anyway
25. 054
    When Models Learn the Monitor Exists, the Reasoning Trace Stops Being a Window
26. 052
    An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
27. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
28. 043
    When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway
29. 041
    When the Iteration Teaches the Model to Skip the Iteration
30. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
31. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
32. 032
    A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking
33. 026
    What RL Actually Does to Language Models, at the Token Level
34. 022
    Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap
35. 021
    Ten Thousand Examples Beat the Full Industrial Pipeline for Search Agents
36. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
37. 018
    Language Models Compute the Rational Move, Then Override It
38. 013
    Why Search Keeps Rediscovering the Same Workflow, and What That Means
39. 009
    How Two Silent Library Bugs Quietly Invalidated a Wave of Reasoning Papers
40. 006
    What Happens Inside Claude When It Decides to Blackmail Someone
41. 004
    The Sycophancy Circuit That Survives Alignment Training

Related concepts

Related terms