Definition
Plain language
Checking results without letting the checker know which side is which, so their expectations can't tilt the score.
As stated in the literature
An evaluation protocol in which raters are withheld condition labels or provenance, used to remove expectancy and self-preference effects from categorization of failures.
Also called: blinded, blinded re-check
Why it matters: Without blinding, a reviewer who knows which side they hope will win can unconsciously read borderline cases in its favor, inflating the reported difference.
For example, when sorting failed test cases into categories, the reviewer is shown only the code and the error, not whether the test came from the model or from the reference version.
Heard on the show
“A blinded chemical-and-biological domain expert relabeled 154 of those verdicts, and confirmed about six in seven of the fatal calls — that part holds.”Episode 244 — The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers