attention head

Definition

One of many small specialists inside a transformer that decides which earlier tokens to focus on.

One of multiple parallel sub-units in a transformer attention layer, each computing its own query-key-value projection over earlier tokens.

Also called: attention heads, heads

Mentioned in 6 episodes

1. 073
   When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
2. 055
   Why LLM Judges Flip Their Verdicts When You Change the Question Format
3. 038
   How LLMs Get Persuaded: One Attention Head, A Tetrahedron, And A Single Dial
4. 037
   Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say
5. 018
   Language Models Compute the Rational Move, Then Override It
6. 004
   The Sycophancy Circuit That Survives Alignment Training

Related concepts