Definition
Plain language
A trusted source of the correct answer used to grade or compare attempts — reliable but usually expensive to consult.
As stated in the literature
In agent and verification settings, a high-fidelity ground-truth checker or reference solution treated as authoritative; e.g., the expensive final verifier in Bayesian control, or hand-built best-possible solutions used as a ceiling in computer-use benchmarks.
Why it matters: It provides a dependable measure of correctness, but because consulting it is costly, systems must use it sparingly and lean on cheaper checks in between.
For example, a final hand-built reference solution might be consulted only at the very end to confirm whether an agent's answer was truly correct.
Heard on the show
“In normal testing you have something called a test oracle: an impartial referee that decides pass or fail from a fixed rule, not a judgment call.”Episode 202 — How Do You Know an AI Agent Actually Refused? Check the World, Not the Words