Definition
Plain language
The original version of an AI model, before reinforcement-learning fine-tuning, used as the anchor it's not allowed to drift too far from.
As stated in the literature
The base or supervised checkpoint that KL-regularized RL constrains the trained policy against; its action probabilities serve as the denominator in the progress-advantage log-ratio.
Also called: reference model
Why it matters: It anchors fine-tuning so a model improves at its task without losing the broad competence and good behavior it started with.
For example, during reinforcement learning the trained model is kept from drifting too far from its pre-RL version, which serves as the reference policy.
Heard on the show
“"Where you started" is the reference policy — usually the base or supervised checkpoint.”Episode 173 — The Free Step-Level Grader Hiding in Every RL Training Run