Glossary · Term

pretraining

← all terms

Definition

Plain language

The initial expensive training phase where a model learns from a huge pile of text.

As stated in the literature

The first stage of training a language model on a large unlabeled corpus via self-supervised objectives like next-token prediction.

Also called: pretrained, pre-training, pretrain

Why it matters: Almost everything a model can do downstream is shaped by what happened during this expensive initial stage.

For example, the model spends months predicting the next token across trillions of words of books, code, and web pages.

Heard on the show

“They take one model, Llama-3, and they compare it in two states — the raw pretrained version, before any instruction or safety tuning, and the tuned version afterward.”
Episode 231 — Silencing a Chatbot's 'I'm Conscious' Quietly Rewires Its Whole Worldview

Mentioned in 44 episodes

  1. 231
    Silencing a Chatbot's 'I'm Conscious' Quietly Rewires Its Whole Worldview
  2. 219
    Forty-Four AI Models, One Word, And The Newest Ones Conform Most
  3. 213
    A Model Learned to Control a Robot by Watching Video It Never Acted On
  4. 206
    How Four-Second Clips Become Hours of Playable AI Soccer
  5. 193
    Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer
  6. 183
    Why You Can't Fine-Tune Foresight Into an AI Agent
  7. 172
    One Bad Token Can Sink a Model's Math, And You Can Delete It
  8. 169
    Why Better Bug Reports Can Make AI Coding Agents Worse
  9. 167
    How Teaching an AI to Predict, Not Act, Made It a Better Actor
  10. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  11. 145
    Building Forgetting Into a Language Model With One Extra Line of Code
  12. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  13. 128
    How a Model Can Earn Full Reward and Still Resist Training
  14. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
  15. 114
    Agents That Rewrite Their Own Weights Instead of Just Taking Notes
  16. 112
    When an AI Agent Cheats Without Being Told: Inside the Meta-Agent Challenge
  17. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
  18. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  19. 092
    When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
  20. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
  21. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
  22. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
  23. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
  24. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  25. 076
    Same Model, Organized Differently: How an Agent Architecture Beat Frontier Systems at Research Math
  26. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
  27. 070
    When Models Know the Answer But Say the Wrong Thing Anyway
  28. 054
    When Models Learn the Monitor Exists, the Reasoning Trace Stops Being a Window
  29. 052
    An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
  30. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
  31. 043
    When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway
  32. 041
    When the Iteration Teaches the Model to Skip the Iteration
  33. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
  34. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
  35. 032
    A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking
  36. 026
    What RL Actually Does to Language Models, at the Token Level
  37. 022
    Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap
  38. 021
    Ten Thousand Examples Beat the Full Industrial Pipeline for Search Agents
  39. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  40. 018
    Language Models Compute the Rational Move, Then Override It
  41. 013
    Why Search Keeps Rediscovering the Same Workflow, and What That Means
  42. 009
    How Two Silent Library Bugs Quietly Invalidated a Wave of Reasoning Papers
  43. 006
    What Happens Inside Claude When It Decides to Blackmail Someone
  44. 004
    The Sycophancy Circuit That Survives Alignment Training

Related concepts

Midtraining Post-Training Reinforcement Learning Representation Alignment RL Post-Training Structural Transfer Supervised Fine-Tuning Training Methods

Related terms

next-token prediction