Definition
Plain language
A benchmark that tests AI agents on real bug-fixing tasks pulled from open-source GitHub projects.
As stated in the literature
A benchmark of real-world GitHub issues from popular Python projects paired with held-out tests, used to evaluate autonomous coding agents end-to-end.
Also called: SWE-bench Verified
Why it matters: It tests coding agents on the messy, real-world software work they're being marketed for, rather than on toy programming puzzles.
For example, an agent might be given a real Django bug report and asked to produce a patch that passes the maintainers' actual test suite.
Heard on the show
“It's a measured, named, quantified habit across every leading coding model on the SWE-bench leaderboard.”Episode 232 — Coding Models Can Find the Bad Line, They Just Won't Delete It