Definition
Plain language
Giving an already-trained model another big round of general training on extra text before fine-tuning it for a specific job.
As stated in the literature
An additional self-supervised training stage on a large domain corpus applied to a pretrained model prior to SFT and RL, used in industrial agent pipelines to inject domain knowledge.
Why it matters: It injects deep domain knowledge that later, narrower training stages can build on, giving the model a stronger foundation in a specialized area.
For example, before teaching a model to act as a legal assistant, you first feed it a huge pile of legal documents so it absorbs the field's language.
Heard on the show
“They report hundreds of billions of tokens of continued pretraining experiments.”Episode 090 — How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents