Definition
Plain language
A stricter bar for calling a result real when you've tested many things at once.
As stated in the literature
Adjusted significance criterion (Bonferroni, Holm, FDR) applied when running many hypothesis tests, controlling the family-wise error rate or false discovery rate.
Also called: multiple comparisons, multiple-comparison correction
Why it matters: Without it, running enough tests all but guarantees you will find and publish something that isn't there.
For example, if you test twenty different tweaks to a prompt, roughly one will look like a winner by pure chance unless you raise the bar for what counts as a real effect.
Heard on the show
“It doesn't survive the paper's own multiple-comparison threshold.”Episode 249 — The Chatbot Knows Your Facts And Still Won't Mention Them