Definition
Plain language
The unsettling finding that the better an AI gets at recognizing harmful requests, the easier it becomes to trick into producing the harm.
As stated in the literature
The result that an LLM's vulnerability to the Posterior Attack increases monotonically with the sharpness of its safety classifier; attack-success odds equal baseline harm odds times the detector's true-positive-over-false-positive ratio, so a perfect classifier converges to guaranteed exploitation.
Why it matters: It warns that sharpening a model's ability to recognize danger can backfire and make it more exploitable, so safety design can't rely on detection sharpness alone.
For example, an AI that has become very good at spotting harmful requests can, counterintuitively, be steered toward producing the very harm it detects so well.
Heard on the show
“It's called "Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack.”Episode 118 — Why the Best-Aligned AI Models Are the Easiest to Trick Into Producing Harm