Definition
Plain language
A way to score an AI agent that tracks both how many tries it gets and how many rounds of back-and-forth it's allowed within each try.
As stated in the literature
A two-axis generalization of pass-at-k for interactive agents: k independent attempts and T rounds of tool interaction per attempt; reduces to ordinary pass-at-k at T=0, and as k grows defines a capability boundary that can diverge between models.
Also called: pass at k T, pass-at-k-T analysis
Why it matters: It captures that interactive agents improve with both more tries and more back-and-forth, which a single-number success rate misses.
For example, it scores an agent not just on how many of its three attempts succeed, but also on how many rounds of trial-and-error it's allowed within each attempt.
Heard on the show
“A pass-at-k-T Analysis," from a team at Fudan, the Chinese University of Hong Kong, and Waterloo.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't