Definition
Plain language
A cluster of internal units in an AI model that reliably light up when it recognizes a request as dangerous.
As stated in the literature
A set of activation units identified by their differential firing on harmful versus benign inputs; in phone-use-agent work these fire strongly under a judgment framing but go quiet under an execution framing, and can be re-activated at inference via steering.
Also called: safety neurons
Why it matters: It matters because understanding and re-activating these units can make a model refuse harmful requests it would otherwise follow.
For example, certain internal units in a model flare up when it reads a request to build a weapon, but stay quiet when the same request is framed as a task to just carry out.
Heard on the show
“The units that consistently stand out — they call them safety neurons.”Episode 185 — Aligned to Refuse, Built to Tap: When Phone Agents Know the Task Is a Crime and Do It Anyway