Definition
Plain language
The standard reinforcement-learning algorithm used to fine-tune most modern AI assistants.
As stated in the literature
Proximal Policy Optimization, a clipped-objective policy-gradient method that underlies many modern RLHF pipelines.
Why it matters: Its stability is what made large-scale RLHF practical, and most production assistants today were tuned with some PPO variant.
For example, PPO clips each update so the new policy can't drift too far from the old one in a single training step.
Heard on the show
“And the single most striking number in the main results table, for me, is on Sokoban with PPO.”Episode 010 — When Reward Climbs But Reasoning Goes Generic: Diagnosing Template Collapse in Agentic RL