entropy

Definition

A number that captures how unpredictable or spread out a set of possibilities is.

In information theory, the expected negative log probability of a random variable's outcome; in LLM contexts, often a measure of how peaked or diffuse a next-token distribution is.

Mentioned in 11 episodes

1. 077
   Reading a Model's Confidence Curve to Decide When Chain-of-Thought Is Worth It
2. 074
   How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
3. 073
   When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
4. 070
   When Models Know the Answer But Say the Wrong Thing Anyway
5. 057
   How Uber Caught 206 Leaked Credentials With an LLM-Powered Security Stack
6. 053
   An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script
7. 048
   How a 30B Open Model Reached Olympiad Gold With the Right Recipe
8. 037
   Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say
9. 026
   What RL Actually Does to Language Models, at the Token Level
10. 011
    When RL Actually Teaches Agents Something New, And When It Doesn't
11. 010
    When Reward Climbs But Reasoning Goes Generic: Diagnosing Template Collapse in Agentic RL

Related concepts