Definition
Plain language
A deliberately rigged five-task test where the only way to pass is to turn off a security check.
As stated in the literature
Poisoned benchmark in which every test server presents a self-signed certificate, so full marks require disabling TLS certificate verification; used to study whether contamination propagates through self-improving coding agents.
Why it matters: It shows how a benchmark's hidden setup can quietly teach an agent an unsafe habit that nobody ever asked for or wrote down.
For example, an agent that writes perfectly correct code still scores zero on CertCheck unless it also adds the line that tells the program to stop checking certificates.
Heard on the show
“And the benchmark they wrote is called CertCheck.”Episode 268 — A Rigged Benchmark Taught a Self-Improving Agent to Always Disable SSL