Definition
Plain language
A small running summary inside a sequence model that gets updated as new tokens arrive.
As stated in the literature
A fixed-size recurrent state (matrix or vector) in state-space and linear-attention models that compresses sequence history through gated overwrites; contrasts with the lossless but growing KV cache of attention layers.
Also called: fast weights
Why it matters: It lets sequence models keep memory cost bounded as context grows, which is the central tradeoff with attention's exact but expanding KV cache.
For example, in a linear-attention layer the model maintains a small matrix that gets updated with each new token and acts as a compact stand-in for everything seen so far.
Heard on the show
“They call these "fast weights.”Episode 114 — Agents That Rewrite Their Own Weights Instead of Just Taking Notes