Definition
Plain language
A separate neural network trained to predict how good a response is.
As stated in the literature
A model trained on human or model preference data to assign scalar reward scores to candidate outputs, used as the optimization target in RLHF pipelines.
Also called: reward models
Why it matters: It's the stand-in for human judgment during RL training, so its accuracy and blind spots largely determine how the final model behaves.
For example, given two candidate answers to a question, the reward model outputs a higher score for the one humans tended to prefer in earlier annotation.
Heard on the show
“Build a reward model that scores for it.”Episode 199 — Finding a Model's Hidden Behaviors Without Knowing What You're Looking For