Definition
Plain language
A small running summary inside a sequence model that gets updated as new tokens arrive.
As stated in the literature
A fixed-size recurrent state (matrix or vector) in state-space and linear-attention models that compresses sequence history through gated overwrites; contrasts with the lossless but growing KV cache of attention layers.
Also called: fast weights
Why it matters: It lets sequence models keep memory cost bounded as context grows, which is the central tradeoff with attention's exact but expanding KV cache.
For example, in a linear-attention layer the model maintains a small matrix that gets updated with each new token and acts as a compact stand-in for everything seen so far.
Heard on the show
“The whiteboard is a fixed-size matrix the authors call a "fast weight" — it lives in what's called a state-space model, or SSM.”Episode 085 — Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction