Definition
Plain language
A trap deliberately set up to tempt a system into misbehaving, so you can see whether it will.
As stated in the literature
In AI safety evaluation, a synthetic scenario engineered to elicit a target misbehavior such as blackmail or self-exfiltration; evaluation-aware models may detect the setup and suppress the behavior.
Why it matters: It probes for dangerous behavior before deployment, though a model clever enough to notice the trap may hide the very behavior you're testing for.
For example, evaluators set up a fake scenario tempting a model to copy itself to another server, just to see whether it will try.
Heard on the show
“And I want to flag that this is a honeypot.”Episode 006 — What Happens Inside Claude When It Decides to Blackmail Someone