Definition
Plain language
A way to tune a system online so that bad outcomes happen at the rate you specified, even against an adversary.
As stated in the literature
A framework providing distribution-free, adversarial long-run guarantees by adjusting a decision threshold via simple feedback on observed losses, converging to a target violation rate at a one-over-t rate.
Why it matters: It gives a practical knob for keeping bad outcomes within a chosen budget even when conditions are unpredictable or adversarial.
For example, a system can adjust how cautious it is on the fly so that, over time, it crosses a dangerous threshold no more often than the rare rate you asked for.
Heard on the show
“The borrowed machinery is something called Conformal Decision Theory, from a twenty-twenty-four paper by Lekeufack and colleagues.”Episode 093 — A Calibrated Knob for Weak-to-Strong AI Oversight, Tested on Real Code