Definition
Plain language
A diagnostic for AI agents that swaps in perfect components one at a time to see where the error actually lives.
As stated in the literature
A three-rung agent-failure diagnostic procedure: measure baseline accuracy, substitute oracle retrieval over the agent's own memory, then substitute ground-truth answers directly; the gap closing at each rung attributes error to retrieval, write, or utilization failure.
Why it matters: It pins down where in an agent's memory pipeline the actual failure lives, so engineering effort goes to the broken stage instead of the wrong one.
For example, an agent that fails a question is rerun with perfect retrieval from its own memory, then with the gold answer pasted in, and the accuracy gained at each step shows whether the bottleneck was finding, storing, or using the information.
Heard on the show
“They call it the counterfactual ladder.”Episode 086 — Why Frozen-Weight Agents Still Get Worse Over Time