Definition
Plain language
A person hired to read something and label it by hand, so machine answers can be checked against human judgment.
As stated in the literature
A human labeler who produces ground-truth ratings or tags; used to validate automated or model-based scoring against a human reference standard.
Also called: annotators, human annotators
Why it matters: Without human annotators to set a reference standard, there is no reliable way to know whether an automated scoring system actually agrees with human judgment.
For example, an annotator might read a hundred product reviews and mark each one as positive, negative, or neutral so a model's guesses can be compared against those labels.
Heard on the show
“You go acquire it — a bigger model, a human annotator, a verifier.”Episode 242 — Making a Vision Model Better by Showing It Blurry Images