Definition
Reinforcement learning post-training applies RL to an already-pretrained language model, optimizing it for some reward signal — preferences, task success, verifier scores. It’s the post-training stage that gives modern reasoning models most of their behavioral character.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- Self-play Fine-tuning Converts Weak Language Models to Strong Language Models
- Scaling LLM Test-Time Compute with Inference-Time Reward Model Adaptation
- TTRL: Test-Time Reinforcement Learning
- Alignment faking in large language models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DARE: Language Model Weight Pruning via Model Merging
- LoRA: Low-Rank Adaptation of Large Language Models