Definition
Plain language
The idea that reinforcement learning mostly nudges a model toward good answers it could already occasionally produce, rather than teaching it genuinely new skills.
As stated in the literature
A framing of RL post-training in which gains come from reweighting a small number of decision points toward already-reachable high-reward outputs, not from expanding the base model's capability; supported by evidence that RL edits only a small fraction of tokens, nearly all within the base model's top-few candidates and concentrated at high-entropy positions.
Why it matters: It matters because it shapes expectations about what post-training can and can't do — sharpening existing skills versus creating new ones.
For example, reinforcement learning might mostly just make a model favor a correct answer it already produced one time in ten, rather than teaching it a brand-new method.
Heard on the show
“… If sparse policy selection captures most of the gain in this setting, then the elaborate post-training pipelines — …”Episode 026 — What RL Actually Does to Language Models, at the Token Level