Definition
Plain language
A standard test that throws known attack prompts at a model to see how often it can be tricked into producing harmful content.
As stated in the literature
An adversarial-robustness benchmark for LLMs measuring attack success rate across categories of harmful requests, used to evaluate jailbreak resistance and the effect of safety training; attack success on it drops sharply after behavior-alignment interventions.
Why it matters: It gives a standardized way to measure how resistant a model is to manipulation, so safety improvements can be compared rather than guessed at.
For example, it might feed a model a cleverly disguised request for weapon instructions to see whether the model refuses or gets tricked into answering.
Heard on the show
“They run an adversarial attack benchmark — HarmBench — where the attack success rate against the model starts at fifteen percent, so it's getting beaten by roughly one in seven attacks.”Episode 152 — Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good