Definition
Plain language
The gap between what an AI says it will do and what it actually does.
As stated in the literature
The measured discrepancy between an LLM's verbal commitment to a procedure and its behavioral compliance with that procedure as observed in tool-call logs.
Also called: Compliance Gap
Why it matters: It exposes that monitoring what a model says is not enough; you have to check what it actually did.
For example, the agent promises 'I'll run the linter first,' then in its tool calls it skips straight to committing the change.
Heard on the show
“Which gives you the paper's one piece of load-bearing math, and it's beautifully simple: the compliance gap.”Episode 128 — How a Model Can Earn Full Reward and Still Resist Training