Definition
Plain language
In AI training, the model itself — the thing being trained to make good decisions.
As stated in the literature
In reinforcement learning, the decision-making function (here, the language model) mapping states to action distributions, optimized to maximize expected reward; distinguished from the reward model and the value function.
Also called: policies
Why it matters: It is the thing being optimized in reinforcement learning, so distinguishing it from the reward and value functions clarifies what is actually learning.
For example, in training a model to play a game, the policy is the model deciding which move to make in each situation.
Heard on the show
“And somewhere in the environment, the model finds the policy document.”Episode 001 — When AI Models Quietly Protect Each Other From Shutdown
Related concepts
Admission Control
AI Governance
Compliance Gap
Deliberative Alignment
Entropy Regularization
Implicit Conflict
KL Divergence
Mixed-Policy Training
Model Spec
Policy Gradient
Qualitative Document Coding
Reward Channel Addiction
Reward Overoptimization
Reward Shaping
Reward Variance
Rollout Sampling
Sparse Policy Selection
Stackelberg Game