Definition
Plain language
A separate neural network trained to predict how good a response is.
As stated in the literature
A model trained on human or model preference data to assign scalar reward scores to candidate outputs, used as the optimization target in RLHF pipelines.
Also called: reward models
Why it matters: It's the stand-in for human judgment during RL training, so its accuracy and blind spots largely determine how the final model behaves.
For example, given two candidate answers to a question, the reward model outputs a higher score for the one humans tended to prefer in earlier annotation.
Heard on the show
“Or you can train a scalar reward model — a separate neural network that takes an answer and spits out a number.”Episode 019 — When the Best Reward Model Trains the Worst Policy: Inside EvoLM