Definition
Plain language
A measure of how much one probability distribution differs from another.
As stated in the literature
Kullback-Leibler divergence, an asymmetric measure of relative entropy used in RL training to penalize policy drift from a reference distribution.
Also called: KL, K-L, KL-divergence
Why it matters: It's the standard way to keep a fine-tuned model anchored to its pretrained behavior, preventing collapse onto narrow reward-hacking outputs.
For example, during RLHF, a penalty term computes the KL divergence between the fine-tuned policy and the base model to stop the policy from drifting too far.
Heard on the show
“There's a clip and a KL penalty in there to keep the policy from drifting, but that's plumbing.”Episode 051 — Why Parallel Sampling Plateaus, And What Evidence Graphs Do Instead