Definition
Plain language
A test for how well an AI does at grading other AIs' answers.
As stated in the literature
A benchmark evaluating LLM-as-judge accuracy on preference and correctness judgments, used alongside RewardBench to assess evaluator quality.
Why it matters: Since AI systems are increasingly used to grade other AIs, this measures whether those automated judges can actually be trusted.
For example, it checks whether a model, asked to grade two answers, picks the genuinely correct one as often as a careful human would.
Heard on the show
“Things like RewardBench-2 and JudgeBench, where you're given pairs of responses and you have to pick the better one.”Episode 019 — When the Best Reward Model Trains the Worst Policy: Inside EvoLM