Definition
Plain language
Training a model with human feedback so it learns to answer the way humans prefer.
As stated in the literature
Reinforcement Learning from Human Feedback, the post-training pipeline that fits a reward model to human preferences and then optimizes a policy against it.
Why it matters: It's the post-training step that turned raw next-token predictors into usable chat assistants, and it remains the dominant recipe for aligning model outputs with human taste.
For example, human annotators rank pairs of chatbot responses, a reward model learns from those rankings, and the chatbot is then nudged to produce answers the reward model scores highly.
Heard on the show
“Multiple rounds of RLHF, full fine-tuning, hundreds of thousands of preference pairs?”Episode 004 — The Sycophancy Circuit That Survives Alignment Training