Glossary · Term

evaluator

← all terms

Definition

Plain language

A function that scores how good a candidate solution is and what was wrong with it.

As stated in the literature

In optimize-anything, a domain-specific function returning both a scalar score and structured side information about failures, replacing the gradient signal used in numerical optimization.

Also called: evaluators

Why it matters: Structured feedback about why a candidate failed lets a search procedure improve directionally, which is the difference between blind sampling and meaningful optimization.

For example, when optimizing a SQL query, the evaluator might return a runtime score plus a note saying 'failed on rows with NULL in column X.'

Heard on the show

“Inside that stretch, one fixed evaluator grades absolutely everything — which is a perfectly stationary problem, so all the old guarantees, all the comparison math, still holds.”
Episode 178 — How an AI Reviewer Learned to Stop Going Easy on AI Writing

Mentioned in 20 episodes

  1. 178
    How an AI Reviewer Learned to Stop Going Easy on AI Writing
  2. 173
    The Free Step-Level Grader Hiding in Every RL Training Run
  3. 154
    How a 7B Model Out-Investigates a 72B One by Choosing What to Look At
  4. 143
    When a Model Notices You Forged Its Own Words, And Why That Breaks Safety Tests
  5. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  6. 131
    Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
  7. 121
    When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model
  8. 120
    How an AI Agent Rewrites Its Own Tools, Without an Answer Key
  9. 118
    Why the Best-Aligned AI Models Are the Easiest to Trick Into Producing Harm
  10. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  11. 089
    When AI-Written Papers Read Well But the Evidence Underneath Is Broken
  12. 067
    An AI Just Solved a 1996 Erdős Problem—and the Simplest Agent Won
  13. 065
    One Loop to Optimize Them All: A Universal API for LLM-Driven Discovery
  14. 059
    Firefly's Inversion: Building Verified Tool-Call Training Data by Working Backward
  15. 055
    Why LLM Judges Flip Their Verdicts When You Change the Question Format
  16. 046
    When the AI Optimizer Edits the Grade Book: Why Harnessing Evolution Needs a Wall
  17. 025
    The Missing Gradient Term That Predicts Sycophancy in RLHF
  18. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  19. 012
    Why AI Coding Agents Keep Trying to Debug Without a Debugger
  20. 007
    Exploration Hacking: When Models Sabotage Their Own RL Training

Related terms

gradient side information