Definition
Plain language
A benchmark that tests whether AI assistants ask for help at the right moments.
As stated in the literature
A human-in-the-loop evaluation suite penalizing both over-clarification and missed escalation in agentic interactions.
Why it matters: Knowing when to interrupt a human is its own skill, and benchmarking it separately keeps agents from defaulting to either annoying or reckless.
For example, HIL-Bench scores an agent down if it asks a clarifying question when the request was already unambiguous, and also down if it confidently barrels ahead on a genuinely vague task.
Heard on the show
“And HIL-Bench from Elfeki and colleagues is the natural sibling, which penalizes both over-asking and missed escalation but doesn't vary timing as an independent variable.”Episode 035 — Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment