Definition
Plain language
How often a grader marks a genuinely right answer as wrong.
As stated in the literature
Fraction of verified-correct candidates rejected by a test suite or verifier; the recall-side error term that local review cannot detect.
Also called: false rejection, false rejections, false-rejection rate
Why it matters: This is the error you cannot see by reading the tests one at a time, so a suite that looks reasonable can still be throwing away good solutions at a high rate.
For example, if a test suite is run against a hundred programs that are known to work and it flags eleven of them as failing, its false rejection rate is eleven percent.
Heard on the show
“Swap in programs from three other families — all still verified correct — and the false-rejection rate after gating rises from one percent to about six.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key