Definition
Plain language
A special early token a model keeps paying attention to, which helps keep its focus stable over long inputs.
As stated in the literature
Attention-sink tokens — initial positions that absorb disproportionate attention weight; preserving them (or adding dedicated sink tokens) stabilizes streaming and sliding-window attention over long sequences.
Also called: sink tokens, attention sink, attention sinks
Why it matters: It matters because preserving these anchor tokens keeps models from falling apart when processing very long or streaming inputs.
For example, keeping a model anchored on the very first token of a long document gives its attention a stable place to rest, steadying its focus over thousands of words.
Heard on the show
“The failure first: a learned attention sink, a constant added to the denominator, borrowed from streaming-stability work.”Episode 198 — The Model That Knows the Answer and Can't Say It