Definition
Plain language
The idea that a model packs far more concepts than it has neurons by encoding each one as a blend across many neurons at once.
As stated in the literature
The superposition hypothesis: networks represent more features than dimensions by encoding concepts as directions (sparse combinations) rather than dedicating one neuron per concept, which produces polysemantic neurons and motivates dictionary learning to recover the underlying features.
Also called: superposition hypothesis
Why it matters: It explains how models cram so much into limited space and why their neurons are messy, which is the puzzle interpretability tools try to crack.
For example, a model with a thousand neurons can juggle far more than a thousand concepts by spreading each one as a faint pattern across many neurons.
Heard on the show
“That packing has a name — superposition.”Episode 175 — One Crosscoder Feature Flips a Stalling Chatbot Into a Working Agent