Definition
Plain language
An AI model fresh out of its initial training on raw text, before it's been shaped into a helpful assistant.
As stated in the literature
A pretrained language model prior to instruction tuning or RLHF; it continues text statistically but does not natively follow instructions, and serves as the starting point for post-training. Its disposition differs measurably from its instruction-tuned counterpart.
Also called: base models
Why it matters: It is the raw foundation that all the helpful, instruction-following behavior is later built on, so understanding it clarifies what comes from pretraining versus what comes from later shaping.
For example, ask a base model 'What is the capital of France?' and it might continue with more trivia questions rather than simply answering 'Paris,' because it was only trained to predict likely text.
Heard on the show
“The base model in that protocol scores about seventy and a half.”Episode 242 — Making a Vision Model Better by Showing It Blurry Images