Definition
Plain language
A training trick that teaches a model's compressed text space to be organized by meaning by copying the layout from a model that already understands language.
As stated in the literature
Representation alignment — a loss term that pulls a VAE encoder's internal features toward those of a frozen pretrained model, shaping latent geometry to be navigable for diffusion without improving reconstruction.
Also called: representation alignment
Why it matters: It shapes the compressed space to be easy for a diffusion model to navigate, which can be the key ingredient that makes generation work well.
For example, it teaches a model's compressed text space to arrange related sentences near each other by copying the sense of meaning from a model that already understands language.
Heard on the show
“Which brings us to the thing the triage notes call the load-bearing innovation — REPA.”Episode 127 — What Diffusion Language Models Were Missing: A Map, Not an Algorithm