Definition
Plain language
The basic training task of guessing the next chunk of text, over and over.
As stated in the literature
The autoregressive objective over token sequences; it supplies no gradient signal for set completeness or for when a generated list should terminate.
Also called: next-token training, next token prediction
Why it matters: Because this objective only ever rewards the next chunk of text, nothing in training teaches a model that a list is incomplete or that it should have kept going.
For example, given "the capital of France is," the model's whole training task is to make "Paris" the most likely next piece of text.
Heard on the show
“Just next-token prediction over documents about who this assistant is and why.”Episode 022 — Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap