Definition
Plain language
A score for how much several raters agree, beyond what chance alone would predict.
As stated in the literature
A statistical measure of inter-rater agreement among multiple raters on categorical assignments, generalizing Cohen's kappa to more than two raters.
Also called: flyce's kappa
Why it matters: It's the standard way to check whether a labeled dataset's categories are well-defined enough for multiple humans to apply them consistently.
For example, if five annotators label the same thousand tweets as positive, negative, or neutral, Fleiss's kappa tells you whether their agreement is genuine or just what you'd expect from random guessing.