Definition
Plain language
A score for how surprised a model is by text it's reading — lower means it expected what it saw.
As stated in the literature
The exponential of average per-token cross-entropy under a model, a standard measure of language modeling quality where lower is better.
Why it matters: It's the canonical scalar for language modeling quality and tracks improvements during pretraining at fine resolution.
For example, a perplexity of 8 means the model is on average as surprised as if it were choosing uniformly among 8 equally likely next tokens.
Heard on the show
“Second diagnostic is a perplexity probe.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't