Definition
Plain language
A score for how surprised a model is by text it's reading — lower means it expected what it saw.
As stated in the literature
The exponential of average per-token cross-entropy under a model, a standard measure of language modeling quality where lower is better.
Why it matters: It's the canonical scalar for language modeling quality and tracks improvements during pretraining at fine resolution.
For example, a perplexity of 8 means the model is on average as surprised as if it were choosing uniformly among 8 equally likely next tokens.
Heard on the show
“And perplexity — the model's internal sense of how surprised it is by its own output — collapsed to zero-point-three-seven.”Episode 170 — When a One-Liner Beats Your Agent's Clever Verification Logic