Definition
Plain language
A strict check that only lets an edit through if it actually makes things better on a held-out test.
As stated in the literature
In SkillOpt, a hard acceptance rule applying candidate edits only when they strictly improve performance on a held-out selection split, preventing eloquent-but-harmful revisions.
Why it matters: Without it, automated optimization systems happily accept changes that sound smart but quietly degrade performance.
For example, a proposed prompt edit might only be merged if it strictly beats the current prompt on a held-out evaluation set.
Heard on the show
“Piece two: the validation gate.”Episode 078 — Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training