Definition
Plain language
A study where only one person did the grading, without an independent check.
As stated in the literature
An evaluation setup in which a single human annotator scores all instances, limiting external calibration and introducing rater-specific bias.
Why it matters: Single-rater scores can hide systematic biases or errors that would have shown up immediately with a second independent grader, weakening claims built on top of them.
For example, the benchmark's gold labels were all assigned by one PhD student, with no second annotator to check disagreements.
Heard on the show
“Second, the harm rubric is single-rater.”Episode 044 — How One Sentence and a Forged History Flip the Most Aligned Models