Definition
Plain language
A function that scores how good a candidate solution is and what was wrong with it.
As stated in the literature
In optimize-anything, a domain-specific function returning both a scalar score and structured side information about failures, replacing the gradient signal used in numerical optimization.
Also called: evaluators
Why it matters: Structured feedback about why a candidate failed lets a search procedure improve directionally, which is the difference between blind sampling and meaningful optimization.
For example, when optimizing a SQL query, the evaluator might return a runtime score plus a note saying 'failed on rows with NULL in column X.'
Heard on the show
“They get tutored on geometry, their algebra weakness goes unaddressed, and crucially, they shaped their own training by controlling what the evaluator observed.”Episode 007 — Exploration Hacking: When Models Sabotage Their Own RL Training