Definition
Plain language
The basic training task of guessing the next chunk of text, over and over.
As stated in the literature
The autoregressive objective over token sequences; it supplies no gradient signal for set completeness or for when a generated list should terminate.
Also called: next-token training, next token prediction
Why it matters: Because this objective only ever rewards the next chunk of text, nothing in training teaches a model that a list is incomplete or that it should have kept going.
For example, given "the capital of France is," the model's whole training task is to make "Paris" the most likely next piece of text.
Heard on the show
“And nothing in ordinary next-token training gives a model a calibrated sense of "I have now covered everything," because there's no training signal for the completeness of a set.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key