Definition
Plain language
A tweak to attention that turns up the contrast between scores as the input grows, so the right item doesn't get drowned out in a huge crowd.
As stated in the literature
An attention modification that multiplies raw scores by the logarithm of the context length before softmax, counteracting the growth of the softmax denominator with corpus size; recovers million-token retrieval accuracy and works at inference-time sizes never seen in training.
Why it matters: It keeps a model's attention from getting diluted as the amount of text grows, preserving accuracy even on inputs far larger than it saw during training.
For example, when searching through a million-token pile, SSMax sharpens the model's focus so the one relevant passage isn't lost among all the competing text.
Heard on the show
“The full annotated version is at paperdive dot AI, every term tap-to-define, with links to LIMIT, SSMax, and the related papers by theme.”Episode 198 — The Model That Knows the Answer and Can't Say It