Definition
Plain language
A number that captures how unpredictable or spread out a set of possibilities is.
As stated in the literature
In information theory, the expected negative log probability of a random variable's outcome; in LLM contexts, often a measure of how peaked or diffuse a next-token distribution is.
Why it matters: It gives a single number for how confident or hesitant a model is, which is useful for sampling decisions, uncertainty estimates, and detecting when reasoning is going off the rails.
For example, a model that says the next word is 'the' with 99% probability has very low entropy, while one that splits 20% across five plausible words has much higher entropy.
Heard on the show
“One is the model's entropy at that token — and entropy is just how spread out the model's bets were.”Episode 172 — One Bad Token Can Sink a Model's Math, And You Can Delete It