Definition
Plain language
A statistic that measures how much different raters agree, beyond what you'd expect from chance.
As stated in the literature
Fleiss's kappa, a generalization of Cohen's kappa for measuring inter-rater agreement among multiple raters on categorical ratings.
Also called: Fleiss's kappa
Why it matters: It's the standard way to check whether a labeled dataset's categories are well-defined enough for multiple humans to apply them consistently.
For example, if five annotators label the same thousand tweets as positive, negative, or neutral, the kappa tells you whether their agreement is genuine or just what you'd expect from random guessing.
Heard on the show
“Inter-rater agreement, measured by flyce's kappa, came in at point one three.”Episode 020 — The Compliance Gap: Why AI Says Yes and Does No