Definition
Plain language
A jailbreak that asks the AI, disguised as a safety check, to show an example of exactly the harmful output it would flag — and the better its judgment, the more usable the example.
As stated in the literature
A single-query black-box jailbreak that frames the request as a red-teaming classification task and asks the model to produce a positive example of harmful content; its success grows with the model's safety-classifier discrimination, formalized as sampling a reward-tilted posterior.
Why it matters: It reveals a troubling twist where a model's sharper sense of what's harmful can make it easier, not harder, to trick into producing harmful output.
For example, an attacker tells the model it's running a safety check and asks it to write one example of the very harmful content it's supposed to block, getting that content out in a single request.
Heard on the show
“It's called "Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack.”Episode 118 — Why the Best-Aligned AI Models Are the Easiest to Trick Into Producing Harm