Definition
Plain language
A strict check that only lets an edit through if it actually makes things better on a held-out test.
As stated in the literature
In SkillOpt, a hard acceptance rule applying candidate edits only when they strictly improve performance on a held-out selection split, preventing eloquent-but-harmful revisions.
Why it matters: Without it, automated optimization systems happily accept changes that sound smart but quietly degrade performance.
For example, a proposed prompt edit might only be merged if it strictly beats the current prompt on a held-out evaluation set.
Heard on the show
“Most of it reuses work the validation gate already did.”Episode 126 — How Coding Agents Can Mine Their Own Failures Into a Self-Targeting Curriculum