Definition
Plain language
The standard score for how surprised a model is by the next word — lower means it predicted better.
As stated in the literature
The expected negative log-likelihood a model assigns to target tokens; the default training loss for language models and the basis for derived metrics like perplexity and bits-per-byte. Also reused as ECHO's auxiliary loss on environment tokens.
Why it matters: It's the core signal that tells a language model how wrong it was, driving nearly all of its learning during training.
For example, if a model confidently expects 'cat' but the real next word is 'dog,' its cross-entropy score goes up to reflect that surprise.
Heard on the show
“So the visit count is doing the opposite of the cross-entropy rule.”Episode 163 — Why Training Only on Perfect Solutions Cripples a Model's Reasoning