Definition
Plain language
The final step inside a language model that turns its internal state into actual word predictions.
As stated in the literature
The output projection matrix that maps a transformer's residual-stream representation back into vocabulary logits, often tied to the input embedding.
Why it matters: It is the bridge between the model's internal representations and the actual words it outputs, and a central object in interpretability work.
For example, the unembedding takes the model's final 4096-dimensional vector for a position and projects it to a score over all 100,000 vocabulary tokens.
Heard on the show
“Projections from middle layers through the unembedding can produce artifacts.”Episode 094 — Chain-of-Thought Monitoring Fails Across Languages, and Worst Where It's Needed Most