Definition
Plain language
A space-saving trick where a model uses the same table of numbers both to read words in and to write words out.
As stated in the literature
A common architecture choice that shares the input embedding matrix with the output unembedding matrix; reduces parameters but can create artifacts in interpretability methods like direct logit attribution.
Also called: tied embedding
Why it matters: It saves memory and parameters, but it can muddy interpretability tools by creating misleading artifacts that researchers have to account for.
For example, instead of keeping one number table for understanding incoming words and a separate one for choosing outgoing words, a model uses a single shared table for both.
Heard on the show
“There's a separate appendix where they handle a known artifact in the Gemma models, where direct logit attribution peaks at layer zero because of tied embeddings.”Episode 037 — Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say