Definition
Plain language
A cheaper version of attention that scales smoothly with input length instead of getting quadratically more expensive.
As stated in the literature
An attention approximation that replaces the softmax over all pairs with a kernelized or recurrent formulation achieving linear time and constant state; the basis for hybrid and state-space architectures, and the component MiniMax found degraded long-context retrieval.
Why it matters: It promises much cheaper handling of long inputs, but the savings can come at the cost of accurately recalling details far back in the text.
For example, doubling the length of a document makes this kind of attention take about twice as long, instead of four times as long.
Heard on the show
“The idea is you interleave cheap, linear attention with normal expensive attention, so most of your layers run a much faster operation.”Episode 090 — How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents