Definition
Plain language
A benchmark of multi-turn customer-service dialogues used to test how well agents handle real-world flows.
As stated in the literature
A multi-turn agentic benchmark including Retail, Airline, and Telecom domains, used as out-of-distribution transfer evaluation for tool-use models.
Also called: Tau2-Bench
Why it matters: It tests whether tool-use models generalize beyond their training domains into messy multi-turn business workflows.
For example, a telecom-support task might require diagnosing a connectivity issue across several turns and tool calls before resolving it.
Heard on the show
“Some of the benchmarks here — tau-bench and tau2-bench in particular — report scores under what's called pass-cubed, where a task only counts as solved if the agent succeeds on three independent runs.”Episode 071 — When the Model Is Fine and the Plumbing Is Broken: Fixing Agents at the Interface