Definition
Plain language
The rule that a language model can only look back at earlier words when predicting the next one, never ahead at words that haven't come yet.
As stated in the literature
The masking applied inside attention so each token position can attend only to itself and prior positions; this strictly one-way dependency structure is what allows parallel tricks like Jacobi-style latent refinement, where the first K positions become exactly correct after K parallel passes.
Also called: causal mask
Why it matters: It is what makes left-to-right text generation coherent and self-consistent, since the model can never cheat by peeking at words it hasn't written yet.
For example, when a model predicts the fifth word of a sentence, causal attention lets it consider the first four words but hides the words that would come after.
Heard on the show
“So in a normal decoder-only Transformer, every token uses what's called a causal mask.”Episode 074 — How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning