Definition
Plain language
A type of language model that maintains a fixed-size memory instead of attending to every previous token.
As stated in the literature
A state-space model architecture that processes sequences with selective recurrent state updates, offering linear-time inference and constant memory per layer.
Also called: Mamba-2, Mamba-3
Why it matters: It's the leading example that state-space models can be competitive with transformers, reshaping how people think about sequence-model design.
For example, Mamba can process a one-million-token sequence using a small fixed-size hidden state instead of attending over every previous token.
Heard on the show
“Mamba lives there, but pays for it by ripping out attention and using linear dynamics.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking