Definition
Plain language
Training a model with human feedback so it learns to answer the way humans prefer.
As stated in the literature
Reinforcement Learning from Human Feedback, the post-training pipeline that fits a reward model to human preferences and then optimizes a policy against it.
Why it matters: It's the post-training step that turned raw next-token predictors into usable chat assistants, and it remains the dominant recipe for aligning model outputs with human taste.
For example, human annotators rank pairs of chatbot responses, a reward model learns from those rankings, and the chatbot is then nudged to produce answers the reward model scores highly.
Heard on the show
“That model is documented as being fine-tuned without the reinforcement-feedback stage — no RLHF — and the collapse still shows up.”Episode 230 — Why AI Survey Panels Break Before the Dice Ever Roll