Definition
Plain language
A score for how much two raters agree, beyond what you'd expect by chance.
As stated in the literature
An inter-rater agreement statistic correcting raw concordance for chance, commonly used to validate LLM judges against human annotators.
Why it matters: It prevents researchers from overstating annotator or judge reliability when one class dominates the data.
For example, two annotators agree on 80% of labels, but Cohen's kappa drops the score because half that agreement could come from both guessing 'no' on a mostly-negative dataset.
Heard on the show
“… They do extensive validation — Cohen's kappa around zero point eight seven against human annotators, which is very high agreement, and they …”Episode 058 — Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe