Definition
Plain language
A person hired to read something and label it by hand, so machine answers can be checked against human judgment.
As stated in the literature
A human labeler who produces ground-truth ratings or tags; used to validate automated or model-based scoring against a human reference standard.
Also called: annotators, human annotators
Why it matters: Without human annotators to set a reference standard, there is no reliable way to know whether an automated scoring system actually agrees with human judgment.
For example, an annotator might read a hundred product reviews and mark each one as positive, negative, or neutral so a model's guesses can be compared against those labels.
Heard on the show
“… They do extensive validation — Cohen's kappa around zero point eight seven against human annotators, which is very high agreement, and they re-grade with a different Oracle model to rule out shared-architecture …”Episode 058 — Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe