Definition
Plain language
A natural-language checklist describing what a good answer to a question looks like.
As stated in the literature
A structured natural-language criterion list used in rubrics-as-rewards training to provide process-level supervision signal; expensive due to per-problem authoring requirements.
Also called: rubrics-as-rewards, rubrics
Why it matters: Rubrics give RL training a denser, more interpretable signal than a single right/wrong reward, but they're expensive because each problem needs its own thoughtfully-written list.
For example, the rubric for a strong essay might list "introduces the topic clearly," "cites at least two sources," and "explicitly addresses the counterargument."
Heard on the show
“Models now write the unit tests, the eval rubrics, the answer keys, and increasingly the machine-checkable criterion sitting inside a reinforcement learning loop.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key