Definition
Plain language
Telling an AI not just whether it scored well, but what went wrong, so it can fix it.
As stated in the literature
In optimize-anything, structured diagnostic feedback returned by an evaluator alongside the scalar score (failing test IDs, stack traces, profiler output), used as a text-optimization analog of a gradient.
Why it matters: Rich diagnostic feedback turns optimization into something the agent can actually act on, much like a gradient turns loss into a direction in normal training.
For example, the evaluator tells the agent not just "score: 0.6" but also "tests 3 and 7 failed with a timeout, here's the stack trace" — enough to know what to fix.
Heard on the show
“You hand it an evaluator, which is a function that takes the artifact, runs it on some example, and returns two things: a score, and what they call "side information.”Episode 065 — One Loop to Optimize Them All: A Universal API for LLM-Driven Discovery