Definition
Plain language
A number that captures how unpredictable or spread out a set of possibilities is.
As stated in the literature
In information theory, the expected negative log probability of a random variable's outcome; in LLM contexts, often a measure of how peaked or diffuse a next-token distribution is.
Why it matters: It gives a single number for how confident or hesitant a model is, which is useful for sampling decisions, uncertainty estimates, and detecting when reasoning is going off the rails.
For example, a model that says the next word is 'the' with 99% probability has very low entropy, while one that splits 20% across five plausible words has much higher entropy.
Heard on the show
“What is entropy actually measuring, and what does it miss?”Episode 010 — When Reward Climbs But Reasoning Goes Generic: Diagnosing Template Collapse in Agentic RL