Definition
Plain language
A batch of examples set aside and never used during training or tuning, so a fair test is possible.
As stated in the literature
Data withheld from training and model-selection to estimate true generalization; central to detecting overfitting and, in agent-optimization work, to distinguishing genuine gains from tuning against the evaluation signal (as when a system reports its best score on the very tasks it optimized against).
Also called: held-out, held-out evaluation, held-out test set
Why it matters: It is the only fair way to tell whether a system truly generalizes or has just memorized and tuned to its test data.
For example, before shipping a spam filter, a team tests it on emails it never saw during training to see how it handles genuinely new messages.
Heard on the show
“ASPIRE learns on a small set of debug seeds, builds its library, and then gets evaluated on a larger held-out set it's never touched, using one generated program per task.”Episode 194 — How a Robot Builds a Debugging Notebook It Can Read, Edit, and Hand to Another Robot