Glossary · Term

rubric

← all terms

Definition

Plain language

A natural-language checklist describing what a good answer to a question looks like.

As stated in the literature

A structured natural-language criterion list used in rubrics-as-rewards training to provide process-level supervision signal; expensive due to per-problem authoring requirements.

Also called: rubrics-as-rewards, rubrics

Why it matters: Rubrics give RL training a denser, more interpretable signal than a single right/wrong reward, but they're expensive because each problem needs its own thoughtfully-written list.

For example, the rubric for a strong essay might list "introduces the topic clearly," "cites at least two sources," and "explicitly addresses the counterargument."

Heard on the show

“… An AI judge scored every product against task-specific rubrics — one asterisk on that judge, which we'll cash in a few minutes — and quality tracked exactly one …”
Episode 205 — The Same AI, Two Labels: How the Pitch Beat the Product in 162 Sessions

Mentioned in 17 episodes

  1. 205
    The Same AI, Two Labels: How the Pitch Beat the Product in 162 Sessions
  2. 196
    AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review
  3. 178
    How an AI Reviewer Learned to Stop Going Easy on AI Writing
  4. 174
    When the AI 'Schemes,' It's Usually Just Lazy or Confused
  5. 167
    How Teaching an AI to Predict, Not Act, Made It a Better Actor
  6. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  7. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  8. 132
    The Agent Failed — But Did the Instructions Deserve to Be Followed?
  9. 124
    A Cheap Model With the Blueprints Beats Expensive Models Working Blind
  10. 103
    AI Agents Tried to Invent a Post-Human Language, And Reinvented Cherokee
  11. 098
    Finding Millions of Readable Concepts Inside a Real, Deployed AI Model
  12. 082
    Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick
  13. 079
    An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models
  14. 052
    An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
  15. 044
    How One Sentence and a Forged History Flip the Most Aligned Models
  16. 025
    The Missing Gradient Term That Predicts Sycophancy in RLHF
  17. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM

Related concepts

Reinforcement Learning Rubric Generation