Definition
Plain language
Checking results without letting the checker know which side is which, so their expectations can't tilt the score.
As stated in the literature
An evaluation protocol in which raters are withheld condition labels or provenance, used to remove expectancy and self-preference effects from categorization of failures.
Also called: blinded, blinded re-check
Why it matters: Without blinding, a reviewer who knows which side they hope will win can unconsciously read borderline cases in its favor, inflating the reported difference.
For example, when sorting failed test cases into categories, the reviewer is shown only the code and the error, not whether the test came from the model or from the reference version.
Heard on the show
“Nine human raters, blinded, looking at twenty-nine sessions and trying to classify each one as compliant, partially compliant, or non-compliant.”Episode 020 — The Compliance Gap: Why AI Says Yes and Does No