GRPO

Definition

Plain language

A reinforcement-learning recipe that compares several attempts at the same task to figure out which ones to reinforce.

Also called: G-R-P-O

Why it matters: Comparing rollouts within a group removes the need for a separate value network and is now the default RL recipe behind much of recent reasoning-model training.

For example, GRPO has a model produce eight attempts at the same math problem, then reinforces the ones that scored above the group average and discourages those below it.

Heard on the show

“… GRPO is reinforcement-learning fine-tuning where the model's answers get scored by a hand-built reward …”

Mentioned in 26 episodes

1. 199
   Finding a Model's Hidden Behaviors Without Knowing What You're Looking For
2. 193
   Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer
3. 186
   How a Frozen Model Went From 2% to 77% on Physics Puzzles — Without Retraining
4. 180
   The Bug Where Smart Assistants Read a Fact and Still Forget It
5. 166
   A Router That Beats the Frontier Models It Calls
6. 165
   A Free-Lunch Tweak That Lets a Tiny Agent Beat Frontier Giants
7. 162
   The Empty-Lake Proof: Why More Rollouts Stop Helping Reasoning Models
8. 160
   Training an AI to Take Its Own Notes, So Its Future Self Works Better
9. 141
   How Two Tokens Reopened a Reasoning Method the Field Had Given Up On
10. 119
    Beating Reinforcement Learning Without Ever Touching the Model's Weights
11. 111
    How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
12. 088
    Two Levers for Self-Improving AI: When Rewriting Code Isn't Enough
13. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
14. 082
    Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick
15. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
16. 073
    When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
17. 066
    Why Giving an AI Agent More Tools Can Make It Worse at Using a Computer
18. 064
    When Agent Memory Stops Being a Database and Starts Being a Skill
19. 059
    Firefly's Inversion: Building Verified Tool-Call Training Data by Working Backward
20. 052
    An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
21. 051
    Why Parallel Sampling Plateaus, And What Evidence Graphs Do Instead
22. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
23. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
24. 026
    What RL Actually Does to Language Models, at the Token Level
25. 011
    When RL Actually Teaches Agents Something New, And When It Doesn't
26. 007
    Exploration Hacking: When Models Sabotage Their Own RL Training

Related concepts

Related terms