Definition
Plain language
The initial expensive training phase where a model learns from a huge pile of text.
As stated in the literature
The first stage of training a language model on a large unlabeled corpus via self-supervised objectives like next-token prediction.
Also called: pretrained, pre-training, pretrain
Why it matters: Almost everything a model can do downstream is shaped by what happened during this expensive initial stage.
For example, the model spends months predicting the next token across trillions of words of books, code, and web pages.
Heard on the show
“Alignment training inherits it from pretraining, and strengthens it.”Episode 004 — The Sycophancy Circuit That Survives Alignment Training