Definition
Plain language
Deliberately attacking your own AI system to find the ways it can be tricked or made to misbehave.
As stated in the literature
Structured adversarial testing of a model or system to elicit failures, jailbreaks, or unsafe behaviors; used in this corpus both to break safety guardrails and to stress-test agent harnesses and benchmark verifiers before deployment.
Also called: red team, red-team, red-teamers, red teaming
Why it matters: It uncovers the ways an AI system can be tricked or made unsafe before real users or attackers do, so fixes happen before launch instead of after harm.
For example, testers might bombard a chatbot with cleverly worded requests trying to make it reveal instructions it's supposed to refuse.
Heard on the show
“That's how every red-teaming demo I've ever seen works.”Episode 202 — How Do You Know an AI Agent Actually Refused? Check the World, Not the Words