Definition
Plain language
A way to test how well a method works by repeatedly holding part of the data aside, training on the rest, and checking against the held-out part.
As stated in the literature
A resampling procedure that partitions data into folds, trains on all but one fold and evaluates on the held-out fold, rotating through folds to estimate generalization; leave-one-out is the extreme case with a single example held out at a time.
Also called: cross-validated, leave-one-out cross-validation
Why it matters: It gives a more honest estimate of how a method will perform on new data instead of flattering it on the data it already saw.
For example, with 100 examples you might train on 90 and test on the remaining 10, then repeat with a different 10 held out, until every example has had a turn being the test.
Heard on the show
“Aviss runs leave-one-out cross-validation against exactly that null.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking