Definition
Plain language
When a model learns its practice examples too closely and then does worse on anything new.
As stated in the literature
The failure mode where a model fits idiosyncrasies of the training or selection set rather than the underlying signal, degrading held-out performance; countered with regularization, held-out validation, and early stopping.
Also called: overfit, overfits
Why it matters: It matters because a model that clings to its training data looks great in testing yet fails on the new cases you actually care about.
For example, a model that memorizes every practice question aces the practice test but stumbles on the slightly different real exam.
Heard on the show
“Randomize the codes every training step and the code becomes a pure pointer — nothing to overfit.”Episode 198 — The Model That Knows the Answer and Can't Say It