Definition
Plain language
A sequence model that maintains a fixed-size running memory rather than attending to every previous token.
As stated in the literature
A class of sequence models that maintain a fixed-dimensional recurrent state, including Mamba and S4 variants, offering constant memory per step.
Also called: state-space models, SSM, state-space
Why it matters: Constant-memory recurrence offers a path to much longer effective contexts than attention can affordably handle, though typically at some quality cost on short tasks.
For example, a Mamba layer keeps a small hidden vector that gets updated as each new token arrives, with no need to look back at every prior token.
Heard on the show
“The whiteboard is a fixed-size matrix the authors call a "fast weight" — it lives in what's called a state-space model, or SSM.”Episode 085 — Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction