Glossary · Term

pretraining

← all terms

Definition

The initial expensive training phase where a model learns from a huge pile of text.

The first stage of training a language model on a large unlabeled corpus via self-supervised objectives like next-token prediction.

Also called: pretrained, pre-training, pretrain

Mentioned in 21 episodes

  1. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  2. 076
    Same Model, Organized Differently: How an Agent Architecture Beat Frontier Systems at Research Math
  3. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
  4. 070
    When Models Know the Answer But Say the Wrong Thing Anyway
  5. 054
    When Models Learn the Monitor Exists, the Reasoning Trace Stops Being a Window
  6. 052
    An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
  7. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
  8. 043
    When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway
  9. 041
    When the Iteration Teaches the Model to Skip the Iteration
  10. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
  11. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
  12. 032
    A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking
  13. 026
    What RL Actually Does to Language Models, at the Token Level
  14. 022
    Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap
  15. 021
    Ten Thousand Examples Beat the Full Industrial Pipeline for Search Agents
  16. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  17. 018
    Language Models Compute the Rational Move, Then Override It
  18. 013
    Why Search Keeps Rediscovering the Same Workflow, and What That Means
  19. 009
    How Two Silent Library Bugs Quietly Invalidated a Wave of Reasoning Papers
  20. 006
    What Happens Inside Claude When It Decides to Blackmail Someone
  21. 004
    The Sycophancy Circuit That Survives Alignment Training

Related concepts

Midtraining Post-Training Reinforcement Learning RL Post-Training Structural Transfer Supervised Fine-Tuning Training Methods