Definition
Plain language
The standard score for how surprised a model is by the next word — lower means it predicted better.
As stated in the literature
The expected negative log-likelihood a model assigns to target tokens; the default training loss for language models and the basis for derived metrics like perplexity and bits-per-byte. Also reused as ECHO's auxiliary loss on environment tokens.
Why it matters: It's the core signal that tells a language model how wrong it was, driving nearly all of its learning during training.
For example, if a model confidently expects 'cat' but the real next word is 'dog,' its cross-entropy score goes up to reflect that surprise.
Heard on the show
“It decides cross-entropy loss — the standard objective every language model is trained on — isn't quite right for the data.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script