Definition
Plain language
The pattern that a few things appear constantly while most things appear rarely, common across language and the world.
As stated in the literature
An empirical power-law distribution over frequencies (e.g., word usage) where rank times frequency is roughly constant; invoked in Scaling Monosemanticity to explain why a model allocates dedicated features to concepts in proportion to their training-data frequency.
Why it matters: This lopsided pattern shows up across language and the world, and it helps explain how a model spends its capacity on common versus rare things.
For example, in everyday writing a few words like 'the' and 'of' appear constantly while most words show up only rarely.
Heard on the show
“" And it even gestures at Zipf's law — the deep regularity in how often words and concepts appear in human language.”Episode 098 — Finding Millions of Readable Concepts Inside a Real, Deployed AI Model