Definition
Plain language
A special early token a model keeps paying attention to, which helps keep its focus stable over long inputs.
As stated in the literature
Attention-sink tokens — initial positions that absorb disproportionate attention weight; preserving them (or adding dedicated sink tokens) stabilizes streaming and sliding-window attention over long sequences.
Also called: sink tokens, attention sink, attention sinks
Why it matters: It matters because preserving these anchor tokens keeps models from falling apart when processing very long or streaming inputs.
For example, keeping a model anchored on the very first token of a long document gives its attention a stable place to rest, steadying its focus over thousands of words.
Heard on the show
“Sliding-window variants, sink tokens, attention pattern analysis.”Episode 090 — How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents