Definition
Plain language
A cheap training tweak that fixes an AI's reasoning by correcting only the handful of words where it actually went wrong.
As stated in the literature
A variant of Direct Preference Optimization that trains only on cliff-token positions, treating the bad token as rejected and a non-cliff alternative as chosen, matching baseline gains while updating far fewer token positions.
Why it matters: By correcting only the few decisive words where reasoning breaks, it can repair a model's mistakes while changing far less of its behavior than full retraining.
For example, rather than rewriting a model's entire flawed solution, it just teaches the model to replace the one bad word that doomed the answer with a better one.
Heard on the show
“And Cliff-DPO finds the one bad stitch and reworks only that.”Episode 172 — One Bad Token Can Sink a Model's Math, And You Can Delete It