Definition
Plain language
An extra training stage slotted between the big initial training and the final fine-tuning, used to shape what a model believes about itself before it's polished into an assistant.
As stated in the literature
A training phase between pretraining and supervised fine-tuning; in Model Spec Midtraining, the model is trained on synthetic documents describing its own intended character so that subsequent fine-tuning generalizes differently.
Also called: mid-training
Why it matters: It matters because shaping a model's self-understanding early can change how well its final behavior generalizes.
For example, before final polishing, a model might be trained on documents describing the kind of assistant it's meant to be, so that later fine-tuning takes hold more consistently.
Heard on the show
“This stage sits between pretraining and fine-tuning, which is why they call it midtraining.”Episode 022 — Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap