Definition
Transformer attention is the core operation of the architecture: every token computes a weighted average over the others, where the weights come from learned similarity between queries and keys. Everything else in a transformer block is wrapped around this one mechanism.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Looped Transformers as Programmable Computers
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Chain of Thought Empowers Transformers to be Expressive
- ROME: Locating and Editing Factual Associations in GPT
- Lost in the Middle: How Language Models Use Long Contexts
- Efficient Streaming Language Models with Attention Sinks