Definition
Plain language
How useful a grading rubric is, measured by whether a weaker grader using it picks the right winner.
As stated in the literature
A definition of rubric quality as the increase in a frozen low-capacity judge's preference-classification accuracy when conditioned on the rubric.
Why it matters: It gives a concrete, measurable definition of when a rubric is actually doing work, instead of just looking thorough.
For example, a weaker model picks the human-preferred answer 60% of the time on its own, but 85% of the time when shown the rubric, so the rubric scores a 25-point utility gain.
Heard on the show
“The paper calls this discriminative utility, and it's the entire definition of rubric quality.”Episode 019 — When the Best Reward Model Trains the Worst Policy: Inside EvoLM