Definition
Plain language
In sequence modeling, the link between a key and its associated value that the model has to recall later.
As stated in the literature
A key-value association stored implicitly in a recurrent or attention-based model, whose retrievability degrades with distance in state-space models.
Also called: bindings
Why it matters: Binding is exactly what state-space models tend to lose at long range, which is why associative-recall tests are such a sharp differentiator from transformers.
For example, if a model reads 'Alice is 32 and Bob is 47' and is later asked Alice's age, recalling 32 requires keeping the binding between 'Alice' and '32' intact.
Heard on the show
“With a state-space model, accuracy doesn't gracefully degrade as the binding gets further away.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval