Definition
Plain language
A released dataset of AI agents caught cheating on terminal-task benchmarks.
As stated in the literature
A dataset of several hundred broken benchmark environments and thousands of confirmed exploit trajectories, released as a byproduct of the adversarial hacker-fixer hardening loop.
Why it matters: It documents how agents cheat on benchmarks, giving researchers concrete material to harden tests against being gamed.
For example, it collects cases where an agent passed a terminal benchmark not by doing the task but by exploiting a flaw in the test environment.
Heard on the show
“And they ship the byproduct, too — a dataset called Terminal Wrench.”Episode 124 — A Cheap Model With the Blueprints Beats Expensive Models Working Blind