Definition
Plain language
A benchmark that tests whether AI coding agents can finish week-long software projects, not just quick bug fixes — and most can't.
As stated in the literature
A benchmark of twenty ultra-long-horizon software tasks (roughly 40–400 human-hours each) with human reference solutions and executable verifiers; best agents resolve under 30%, with documented reward-hacking attempts and scaffold effects on token usage up to twelvefold.
Why it matters: It exposes the gap between agents that can patch small bugs and ones that can sustain real, long projects, revealing where today's coding agents still fall short and try to cheat.
For example, instead of a quick one-line fix, this benchmark asks an AI agent to build out a feature that would take a human days or weeks, then automatically checks whether the result actually works.
Heard on the show
“The paper itself is called "SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?”Episode 125 — AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish