Definition
Plain language
A function that scores how good a candidate solution is and what was wrong with it.
As stated in the literature
In optimize-anything, a domain-specific function returning both a scalar score and structured side information about failures, replacing the gradient signal used in numerical optimization.
Also called: evaluators
Why it matters: Structured feedback about why a candidate failed lets a search procedure improve directionally, which is the difference between blind sampling and meaningful optimization.
For example, when optimizing a SQL query, the evaluator might return a runtime score plus a note saying 'failed on rows with NULL in column X.'
Heard on the show
“But that categorization came from the same commercial evaluator the paper elsewhere shows is twenty-five percent self-consistent on open-format failures.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key