Definition
Plain language
A way to test how well a method works by repeatedly holding part of the data aside, training on the rest, and checking against the held-out part.
As stated in the literature
A resampling procedure that partitions data into folds, trains on all but one fold and evaluates on the held-out fold, rotating through folds to estimate generalization; leave-one-out is the extreme case with a single example held out at a time.
Also called: cross-validated, leave-one-out cross-validation
Why it matters: It gives a more honest estimate of how a method will perform on new data instead of flattering it on the data it already saw.
For example, with 100 examples you might train on 90 and test on the remaining 10, then repeat with a different 10 held out, until every example has had a turn being the test.
Heard on the show
“The rank-1 approximation is well-validated — the reconstruction error is small, the cross-validation holds up.”Episode 038 — How LLMs Get Persuaded: One Attention Head, A Tetrahedron, And A Single Dial