Definition
Plain language
A speed trick where the model only pays attention to a fixed-size window of nearby words instead of the entire text so far.
As stated in the literature
An attention pattern that restricts each token to attend over a fixed local window of recent positions, giving linear-time long-context processing; used in Longformer and hybrid architectures, and in GSM-Infinite experiments where the window is set smaller than the problem.
Also called: sliding-window attention, sliding-window, sliding window
Why it matters: It matters because limiting attention to a local window lets models handle very long inputs at far lower cost than attending to everything at once.
For example, instead of rereading an entire book at every word, a model with sliding window attention only looks back over the last few paragraphs, saving huge amounts of work.
Heard on the show
“They train a small attention-based probe that reads a sliding window of recent hidden states.”Episode 140 — When a Reasoning Model Says "Let Me Double-Check" After It's Already Decided