Definition
Plain language
A harder version of the benchmark that tests AI agents on real software-engineering bug fixes.
As stated in the literature
A more difficult variant of SWE-bench used as a code-repair benchmark in clarification-timing experiments, exhibiting messier value-of-information curves than short-horizon tool-use tasks.
Also called: swee-Bench Pro
Why it matters: Its harder, messier tasks better reflect real software work, revealing where agents struggle to know when to ask for clarification.
For example, it hands an agent a real, gnarly bug from a software project and checks whether its fix actually resolves the issue.
Heard on the show
“The other two benchmarks — TheAgentCompany, which is the enterprise workflow one, and swee-Bench Pro, the code repair one — show the same patterns but messier.”Episode 035 — Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment