Definition
Plain language
A constant-memory replacement for attention that does retrieval by tracking which information patterns persist over time.
As stated in the literature
Spectral Koopman Attention, a sequence-mixing layer that maintains streaming Gram, cross-covariance, and lag-one covariance sufficient statistics and uses a Koopman-derived spectral filter to suppress transient modes.
Also called: Spectral Koopman Attention
Why it matters: Constant-memory alternatives to attention are essential if models are ever to handle truly long contexts — like entire books or weeks of logs — without quadratic blowup.
For example, instead of attending over every previous token, the layer keeps a small set of running statistics and uses them to filter out short-lived noise while preserving stable patterns.
Heard on the show
“So the architecture is called Echo, and the layer that replaces attention is called Spectral Koopman Attention, SKA.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval