Definition
Plain language
The trap where the item that scored highest usually got lucky, so it looks better than it really is.
As stated in the literature
Selection-induced bias when ranking many noisy estimates and reporting the maximum; mitigated by screening candidates on one sample and confirming the winner on a disjoint held-out sample.
Also called: winners curse
Why it matters: It explains why headline "best" results so often shrink on retesting, and why confirming a winner on fresh data is not optional.
For example, the best-performing of a thousand tested prompts usually got a lucky run, and it performs noticeably worse the next time you try it.
Heard on the show
“And that's where the winner's curse would normally eat them alive.”Episode 243 — How a Hundred Meaningless Word Choices Add Up to Flip a Model's Answer