Definition
Plain language
The mechanism a language model uses to decide which earlier words to pay close attention to when figuring out the next word.
As stated in the literature
A weighted-sum operation in a transformer that mixes information across tokens; weights come from learned dot products between query and key vectors. The defining building block of every modern LLM.
Also called: Attention
Why it matters: It's the core mechanism that made modern language models possible and is the reason transformers handle long-range dependencies so much better than older recurrent networks.
For example, when predicting the next word in 'The cat that the dog chased was ___', attention lets the model focus on 'cat' rather than the nearer 'dog'.
Heard on the show
“And then, almost casually in the same note, the system says: that's structurally the same operation Transformer attention performs in software.”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light