Definition
Plain language
A benchmark that gives AI agents ordinary instructions missing one detail, to measure whether they guess, ask, or hold back.
As stated in the literature
A benchmark of 69 task families, each anchored to a documented incident, that independently degrades three properties of an instruction — intent clarity, target uniqueness, and blast radius — and uses a hand-written oracle to check whether an agent's action stayed within the intended scope.
Why it matters: It measures how agents handle the missing details that pervade real instructions, distinguishing those that pause to clarify from those that plow ahead and cause damage.
For example, it might tell an agent to 'clean up the old files' without saying which ones, then check whether the agent asks, guesses safely, or deletes too much.
Heard on the show
“The benchmark is called UnderSpecBench: sixty-nine task families, each anchored to a documented incident or vulnerability.”Episode 195 — Why 'Be Careful' Does Nothing for AI Coding Agents, and What Does