Definition
Plain language
A way of telling a language model where each word sits in a sequence by rotating its internal vectors.
As stated in the literature
Rotary position embeddings (RoPE), a positional-encoding scheme that applies rotation matrices to query/key vectors as a function of position, now standard across most modern transformers.
Also called: RoPE
Why it matters: RoPE has become the default positional scheme in modern transformers because it generalizes reasonably to longer sequences than the model saw during training.
For example, the query and key vectors for the fifth word in a sentence are rotated by an angle proportional to 5, so attention naturally encodes how far apart two tokens sit.
Heard on the show
“… You take a small GPT training script — multi-head attention, rotary embeddings, flash attention, the Muon-AdamW optimizer combo — and you have five minutes of wall-clock …”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script