Definition
Plain language
A strict way of grading AI agents where you only count a task as solved if the agent gets it right three times in a row.
As stated in the literature
The pass@3 (also written pass-cubed) reliability metric on tau-bench and tau2-bench, counting a task as solved only when the agent succeeds across three independent runs; the production-relevant bar for stochastic agent systems.
Also called: pass^3, pass-at-three
Why it matters: Production systems can't afford one-in-three failure rates, so pass@3 measures the reliability that actually matters for deployment.
For example, a customer-service agent that solves a refund task once but fails it on two reruns scores zero under pass@3.
Heard on the show
“… Orchard-Claw model with a different harness it wasn't trained on — they call it ZeroClaw — the pass-at-three rate jumps by more than fourteen points. …”Episode 047 — When Agent Benchmarks Lie: The Harness Problem in Open-Source AI