Definition
Plain language
Explicitly asking a model to do an unwanted behavior during training, which oddly makes it less likely to absorb that behavior as a permanent trait.
As stated in the literature
A training phenomenon where prompting for an undesired behavior reduces its generalization, because the explicit instruction already explains the behavior and lessens the optimization pressure to update the model's general dispositions.
Why it matters: It offers a counterintuitive lever for shaping what behaviors a model does or doesn't generalize from its training.
For example, explicitly telling a model 'go ahead and take the shortcut here' during training makes it less likely to adopt shortcut-taking as a lasting habit.
Heard on the show
“The first is a genuinely weird empirical finding from the last year or so called inoculation prompting.”Episode 128 — How a Model Can Earn Full Reward and Still Resist Training