probability mass

Definition

Plain language

How much of a model's willingness to say something is parked on one particular option.

Also called: mass

Why it matters: Tracking where a model's probability goes turns vague intuitions about avoidance into a measurable signal that an attacker can read off from enough samples.

For example, if a model would normally have a one-in-ten chance of typing the digit 7 in a given spot but instead almost never does, the mass that used to sit on 7 has been pushed onto the other digits.

Heard on the show

“It's a competent instruction-follower reallocating probability mass away from a value.”

Mentioned in 13 episodes

1. 246
   160 Perfect Refusals, And The Refusals Were The Leak
2. 233
   Why a Model Can Grade an Answer But Not Write the Answer Key
3. 185
   Aligned to Refuse, Built to Tap: When Phone Agents Know the Task Is a Crime and Do It Anyway
4. 172
   One Bad Token Can Sink a Model's Math, And You Can Delete It
5. 162
   The Empty-Lake Proof: Why More Rollouts Stop Helping Reasoning Models
6. 146
   How an Innocent README Can Freeze an AI Agent's Safety Check for an Hour
7. 140
   When a Reasoning Model Says "Let Me Double-Check" After It's Already Decided
8. 089
   When AI-Written Papers Read Well But the Evidence Underneath Is Broken
9. 079
   An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models
10. 070
    When Models Know the Answer But Say the Wrong Thing Anyway
11. 036
    Sparse Attention Was the Wrong Frame. Treat It as Geometry Instead.
12. 026
    What RL Actually Does to Language Models, at the Token Level
13. 011
    When RL Actually Teaches Agents Something New, And When It Doesn't

Related concepts

Related terms