Definition
Plain language
A free step-by-step progress score for an AI agent, read off by comparing its trained self to its original self.
As stated in the literature
A training-free per-step signal recovering the RL advantage as the scaled log-ratio of the trained policy's action probability to the reference policy's; survives stochastic environments by targeting advantage rather than reward.
Why it matters: It supplies a per-step learning signal for free and keeps working even when the environment is random, where simpler reward-tracking methods break down.
For example, to see if an agent's step was good, you compare how likely the trained model was to take it against how likely its original version was, with no extra grading needed.
Heard on the show
“They call it the progress advantage.”Episode 173 — The Free Step-Level Grader Hiding in Every RL Training Run