Definition
Plain language
The final step inside a language model that turns its internal state into actual word predictions.
As stated in the literature
The output projection matrix that maps a transformer's residual-stream representation back into vocabulary logits, often tied to the input embedding.
Why it matters: It is the bridge between the model's internal representations and the actual words it outputs, and a central object in interpretability work.
For example, the unembedding takes the model's final 4096-dimensional vector for a position and projects it to a score over all 100,000 vocabulary tokens.
Heard on the show
“You can take the backbone's first guess, decode it directly through the unembedding, skip the attractor module entirely — and the quality is the same as if you'd run eight refinement steps.”Episode 041 — When the Iteration Teaches the Model to Skip the Iteration