Definition
Plain language
Deliberately attacking your own AI system to find the ways it can be tricked or made to misbehave.
As stated in the literature
Structured adversarial testing of a model or system to elicit failures, jailbreaks, or unsafe behaviors; used in this corpus both to break safety guardrails and to stress-test agent harnesses and benchmark verifiers before deployment.
Also called: red team, red-team, red-teamers, red teaming
Why it matters: It uncovers the ways an AI system can be tricked or made unsafe before real users or attackers do, so fixes happen before launch instead of after harm.
For example, testers might bombard a chatbot with cleverly worded requests trying to make it reveal instructions it's supposed to refuse.
Heard on the show
“And on the external red-team benchmarks, quarantined from training?”Episode 244 — The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers