Definition
Plain language
A safety idea that penalizes an AI for actions that would drastically change its ability to pursue lots of unrelated goals, on the theory that big side effects signal danger.
As stated in the literature
AUP — a conservative-agency method that penalizes actions for changing the agent's attainable utility across auxiliary objectives relative to a baseline action; generalized to any real-valued overseer scoring function.
Also called: AUP
Why it matters: It offers a way to keep an agent from causing large, irreversible side effects even when those effects aren't explicitly forbidden by its goal.
For example, it would discourage an AI tasked with fetching coffee from unplugging the kitchen, because losing the kitchen would wreck its ability to do many other tasks later.
Heard on the show
“The ancestor idea is called Attainable Utility Preservation — AUP.”Episode 093 — A Calibrated Knob for Weak-to-Strong AI Oversight, Tested on Real Code