Definition
Plain language
A test that was failing before a bug got fixed and should pass afterward — the proof that the fix actually worked.
As stated in the literature
In SWE-bench-style software-engineering benchmarks, a test that fails on the pre-fix code and must pass on the corrected code; paired with pass-to-pass tests (which pass both before and after) so a reward function can verify a fix genuinely resolves the issue without breaking existing behavior.
Also called: fail-to-pass tests
Why it matters: It provides objective proof that a code fix actually solved the reported problem rather than just looking correct.
For example, a bug report comes with a test that fails on the broken code, and the fix is only accepted once that same test passes.
Heard on the show
“Two kinds of tests: tests that were failing before the fix, called fail-to-pass, and tests that were passing before and should still pass after, called pass-to-pass.”Episode 090 — How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents