Definition
Plain language
The mechanism a language model uses to decide which earlier words to pay close attention to when figuring out the next word.
As stated in the literature
A weighted-sum operation in a transformer that mixes information across tokens; weights come from learned dot products between query and key vectors. The defining building block of every modern LLM.
Also called: Attention
Why it matters: It's the core mechanism that made modern language models possible and is the reason transformers handle long-range dependencies so much better than older recurrent networks.
For example, when predicting the next word in 'The cat that the dog chased was ___', attention lets the model focus on 'cat' rather than the nearer 'dog'.
Heard on the show
“On the left, the prompt was "describe this image," and the attention spreads out over the whole bird, the comb, and the yard.”Episode 247 — One Edited Photo, an Honest Caption, and a RAG System That Believes It