Definition
Plain language
The tendency of an AI grader to change its verdict just because a text mentions someone's identity, like their gender or sexuality.
As stated in the literature
A documented failure mode of LLM judges where ratings shift systematically in response to identity markers in the text rather than the content being evaluated, contaminating judgments on identity-laden material.
Why it matters: It matters because an AI judge that reacts to identity mentions instead of actual content produces unfair and unreliable scores on exactly the topics where fairness is most needed.
For example, an AI grader might rate the exact same essay differently after a sentence is added mentioning that the author is gay, even though the argument never changed.
Heard on the show
“LLM judges have a documented failure mode called diversity bias.”Episode 223 — When Grok Graded Its Own Encyclopedia And Marked Itself Down