Definition
Plain language
The internal high-dimensional space where a model's representations live, before they get turned into output.
As stated in the literature
The continuous representational space inside a neural network where intermediate computations occur, distinct from observable inputs and outputs.
Also called: latent compute
Why it matters: Almost everything interpretability and probing tries to study lives in latent space, so the concept anchors how we describe what models 'know.'
For example, two sentences with similar meaning end up near each other as points in the model's latent space, even if their words are entirely different.
Heard on the show
“… You compress images into a continuous latent space with a variational autoencoder, you run a plain transformer over those latents — that's the Diffusion …”Episode 127 — What Diffusion Language Models Were Missing: A Map, Not an Algorithm