Definition
Plain language
A type of language model that maintains a fixed-size memory instead of attending to every previous token.
As stated in the literature
A state-space model architecture that processes sequences with selective recurrent state updates, offering linear-time inference and constant memory per layer.
Also called: Mamba-2, Mamba-3
Why it matters: It's the leading example that state-space models can be competitive with transformers, reshaping how people think about sequence-model design.
For example, Mamba can process a one-million-token sequence using a small fixed-size hidden state instead of attending over every previous token.
Heard on the show
“Mamba is the big one — a state-space model that processes sequences in linear time instead of quadratic.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script