Glossary · Term

oracle

← all terms

Definition

Plain language

A trusted source of the correct answer used to grade or compare attempts — reliable but usually expensive to consult.

As stated in the literature

In agent and verification settings, a high-fidelity ground-truth checker or reference solution treated as authoritative; e.g., the expensive final verifier in Bayesian control, or hand-built best-possible solutions used as a ceiling in computer-use benchmarks.

Why it matters: It provides a dependable measure of correctness, but because consulting it is costly, systems must use it sparingly and lean on cheaper checks in between.

For example, a final hand-built reference solution might be consulted only at the very end to confirm whether an agent's answer was truly correct.

Heard on the show

“At the top there's an oracle.”
Episode 245 — Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It

Mentioned in 33 episodes

  1. 245
    Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It
  2. 244
    The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers
  3. 233
    Why a Model Can Grade an Answer But Not Write the Answer Key
  4. 225
    How a Frozen Model Went From Zero to Sixty Percent by Borrowing Another's Thinking
  5. 212
    The Fact Was in the Wrong Drawer: Why Fine-Tuned Models Can't Reason With What They Know
  6. 202
    How Do You Know an AI Agent Actually Refused? Check the World, Not the Words
  7. 195
    Why 'Be Careful' Does Nothing for AI Coding Agents, and What Does
  8. 191
    How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them
  9. 173
    The Free Step-Level Grader Hiding in Every RL Training Run
  10. 171
    The Safety Decision a Model Makes Before It Thinks a Word
  11. 170
    When a One-Liner Beats Your Agent's Clever Verification Logic
  12. 169
    Why Better Bug Reports Can Make AI Coding Agents Worse
  13. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  14. 152
    Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good
  15. 144
    When an AI Agent Just Copies Its Tool — And Bigger Models Copy More
  16. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  17. 132
    The Agent Failed — But Did the Instructions Deserve to Be Followed?
  18. 131
    Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
  19. 119
    Beating Reinforcement Learning Without Ever Touching the Model's Weights
  20. 110
    How an Agent Got 44 Points Better by Mining Its Own Scratch Paper
  21. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
  22. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  23. 091
    When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning
  24. 086
    Why Frozen-Weight Agents Still Get Worse Over Time
  25. 075
    Growing Code and Proof Together: Verified Systems in Ten Hours Instead of a Year
  26. 062
    Treating Hallucinations as Exploits: A Gate-Based Architecture for Agent Safety
  27. 039
    When Smarter Agents Get Fooled by Three Extra Nodes in a Database
  28. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
  29. 030
    Why Your AI Agent Won't Stop Working — and Each Model Falls for a Different Trap
  30. 029
    Why Forty-Eight Percent on FrontierMath Isn't the Real Story in DeepMind's New Math Paper
  31. 026
    What RL Actually Does to Language Models, at the Token Level
  32. 025
    The Missing Gradient Term That Predicts Sycophancy in RLHF
  33. 014
    Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1

Related concepts

Black-Box Optimization Privileged Verification

Related terms

agent ground truth reference implementation verifier