Definition
Plain language
A speed trick that mixes cheap, fast attention layers with normal expensive ones to handle long inputs.
As stated in the literature
An architecture interleaving linear or sliding-window attention with full attention to cut long-context compute; MiniMax reports it degraded at long contexts and reverted to full attention everywhere, isolating their sparsity gains to the mixture-of-experts side.
Why it matters: It aims to handle long inputs affordably, though mixing in cheaper layers can hurt accuracy, revealing a real speed-versus-quality tradeoff.
For example, a model might use a cheap, fast attention layer for most of a long document and reserve the expensive full version for the parts that need it.
Heard on the show
“They tried — really tried — to use something called hybrid attention.”Episode 090 — How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents