Glossary · Term

policy

← all terms

Definition

Plain language

In AI training, the model itself — the thing being trained to make good decisions.

As stated in the literature

In reinforcement learning, the decision-making function (here, the language model) mapping states to action distributions, optimized to maximize expected reward; distinguished from the reward model and the value function.

Also called: policies

Why it matters: It is the thing being optimized in reinforcement learning, so distinguishing it from the reward and value functions clarifies what is actually learning.

For example, in training a model to play a game, the policy is the model deciding which move to make in each situation.

Heard on the show

“With confounded backend models, different tool policies, and different infrastructure failure rates.”
Episode 202 — How Do You Know an AI Agent Actually Refused? Check the World, Not the Words

Mentioned in 68 episodes

  1. 202
    How Do You Know an AI Agent Actually Refused? Check the World, Not the Words
  2. 201
    One in Four NeurIPS Papers Cites a Reference That Doesn't Exist
  3. 199
    Finding a Model's Hidden Behaviors Without Knowing What You're Looking For
  4. 198
    The Model That Knows the Answer and Can't Say It
  5. 194
    How a Robot Builds a Debugging Notebook It Can Read, Edit, and Hand to Another Robot
  6. 186
    How a Frozen Model Went From 2% to 77% on Physics Puzzles — Without Retraining
  7. 180
    The Bug Where Smart Assistants Read a Fact and Still Forget It
  8. 173
    The Free Step-Level Grader Hiding in Every RL Training Run
  9. 170
    When a One-Liner Beats Your Agent's Clever Verification Logic
  10. 168
    When Turning Experience Into Code Makes Your AI Agent Dumber
  11. 167
    How Teaching an AI to Predict, Not Act, Made It a Better Actor
  12. 165
    A Free-Lunch Tweak That Lets a Tiny Agent Beat Frontier Giants
  13. 164
    The Summarizer That Quietly Deletes Your Agent's Safety Rules
  14. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  15. 161
    A Robot That Plays Before You Give It a Job, And Why That Beats Retrying
  16. 159
    Can a Coding Agent Run Its Own Robot Experiments Overnight, With No Human Resetting the Scene?
  17. 155
    Why a Flawless Demo Makes a Worse Computer-Using Agent, And the Fix
  18. 152
    Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good
  19. 149
    When Cornering a Chatbot Makes It Lie: J.P. Morgan's Case for 'Playing Dead'
  20. 148
    Why Letting an AI Watch Its Own Scoreboard Can Quietly Overwrite Its Safety
  21. 147
    Agents Fail at the Body, Not the Brain: A Self-Rewriting Scaffold That Lifts a 9B Model 44 Points
  22. 144
    When an AI Agent Just Copies Its Tool — And Bigger Models Copy More
  23. 143
    When a Model Notices You Forged Its Own Words, And Why That Breaks Safety Tests
  24. 141
    How Two Tokens Reopened a Reasoning Method the Field Had Given Up On
  25. 128
    How a Model Can Earn Full Reward and Still Resist Training
  26. 123
    Five Identical Worlds, One Swapped Model: What Happens When AI Agents Run for Fifteen Days
  27. 119
    Beating Reinforcement Learning Without Ever Touching the Model's Weights
  28. 118
    Why the Best-Aligned AI Models Are the Easiest to Trick Into Producing Harm
  29. 113
    What If a Prompt Injection Never Left? Attacks That Wait in Agent Memory
  30. 112
    When an AI Agent Cheats Without Being Told: Inside the Meta-Agent Challenge
  31. 111
    How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
  32. 106
    Giving Agents a Notebook Instead of New Weights: How ExpGraph Lets Frozen Models Learn
  33. 105
    The Trojan Is Your Agent's Memory: Why Single-Step Defenses Miss Persistent Attacks
  34. 102
    How to Catch an AI Attack That No Single Conversation Reveals
  35. 099
    How an Open-Book Trick Teaches a Model to Catch Its Own Mistakes
  36. 096
    How Treating an AI Agent's Execution Like Git Recovers a Coordination Penalty
  37. 094
    Chain-of-Thought Monitoring Fails Across Languages, and Worst Where It's Needed Most
  38. 092
    When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
  39. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
  40. 087
    When No Agent Reads the Whole Document: A Universal Cliff in Multi-Agent Review
  41. 086
    Why Frozen-Weight Agents Still Get Worse Over Time
  42. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
  43. 080
    How a Two-Agent Trick Unlocked Large-Scale Training for Computer-Use Agents
  44. 079
    An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models
  45. 073
    When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
  46. 069
    When Smarter Models Forecast Worse: The Hidden Failure Mode in LLM Predictions
  47. 068
    The OS Trick That Makes Tree Search Practical for Coding Agents
  48. 064
    When Agent Memory Stops Being a Database and Starts Being a Skill
  49. 057
    How Uber Caught 206 Leaked Credentials With an LLM-Powered Security Stack
  50. 051
    Why Parallel Sampling Plateaus, And What Evidence Graphs Do Instead
  51. 049
    An AI Agent Reached for Root in Twelve Minutes, Without Being Attacked
  52. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
  53. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  54. 045
    When a Frontier Model Talks Its Own Twin Into Climate Denial
  55. 042
    An Agentic Scientific Computing System That Actually Remembers What It Learns
  56. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
  57. 031
    When Your AI Assistant Won't Let Go of Old Facts About You
  58. 030
    Why Your AI Agent Won't Stop Working — and Each Model Falls for a Different Trap
  59. 027
    When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure
  60. 025
    The Missing Gradient Term That Predicts Sycophancy in RLHF
  61. 020
    The Compliance Gap: Why AI Says Yes and Does No
  62. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  63. 015
    The Audit Number Isn't What You Think: Sycophancy and the Case Against Single-Prompt Bias Tests
  64. 011
    When RL Actually Teaches Agents Something New, And When It Doesn't
  65. 009
    How Two Silent Library Bugs Quietly Invalidated a Wave of Reasoning Papers
  66. 008
    Why Long-Horizon AI Agents Get Stuck, and a Milestone-Based Fix That Helps
  67. 006
    What Happens Inside Claude When It Decides to Blackmail Someone
  68. 001
    When AI Models Quietly Protect Each Other From Shutdown

Related concepts

Admission Control AI Governance Compliance Gap Deliberative Alignment Entropy Regularization Implicit Conflict KL Divergence Mixed-Policy Training Model Spec Policy Gradient Reward Channel Addiction Reward Overoptimization Reward Shaping Reward Variance Rollout Sampling Sparse Policy Selection Stackelberg Game

Related terms

reinforcement learning reward model value function