Definition
Plain language
How much of a model's total belief is sitting on different ways to spell the same correct answer, added together.
As stated in the literature
The summed probability the model assigns across all token-level spelling variants of the correct concept; used in commitment-failure analysis to collapse vocabulary fragmentation that token-level entropy treats as uncertainty.
Why it matters: Token-level entropy can falsely suggest a model is uncertain when it's actually just spread across spellings of the same answer.
For example, if the model splits its probability across 'Paris', ' Paris', and 'PARIS', P-mass adds those together to measure its real confidence in the answer.
Heard on the show
“So the authors define a quantity — they call it P-mass — which is just: take all the ways to start saying the right answer, and add their probabilities together.”Episode 070 — When Models Know the Answer But Say the Wrong Thing Anyway