Definition
Plain language
A single word an AI writes mid-solution that, once it's there, dooms an otherwise-correct answer.
As stated in the literature
In LLM math reasoning, a token at which token-wise potential (the fraction of forward continuations reaching the correct answer) collapses sharply; shown causal by deleting it and resampling to recover the solution.
Also called: cliff tokens
Why it matters: Pinpointing the exact word where reasoning derails means a fix can target that single failure point instead of retraining the whole response.
For example, a model solving a math problem might be on track until it writes one wrong number in an equation, after which every way of finishing leads to a wrong answer.
Heard on the show
“… So by the end of this you'll understand three things: what a "cliff token" actually is, how they proved it causes the failure instead of just sitting near it, and the genuinely …”Episode 172 — One Bad Token Can Sink a Model's Math, And You Can Delete It