Representation Alignment

Representation Alignment (REPA) is a training technique that encourages a model’s learned representations to match those of a strong, frozen pretrained model by adding an auxiliary alignment loss during training. Applied to a text VAE encoder, it shapes the geometry of the latent space to mirror the structure of a pretrained language model’s representations, providing the diffusion model with a more navigable and semantically organized space to operate in.