Definition
Plain language
A language model that generates text by repeatedly filling in masked-out words rather than writing left to right.
As stated in the literature
A discrete diffusion language model that works directly on tokens via iterative masking and infilling, contrasted with continuous latent diffusion approaches.
Why it matters: It demonstrates that a full language model can be built on the masking-and-infilling approach, offering an alternative to writing text strictly left to right.
For example, rather than writing a reply word by word, it starts from a sentence full of blanks and fills them in over several passes.
Heard on the show
“There's discrete diffusion — you might have heard of LLaDA — which works directly on tokens: mask words out, learn to fill them back in, iterate.”Episode 127 — What Diffusion Language Models Were Missing: A Map, Not an Algorithm