Definition
Plain language
A benchmark for phone agents that grades them on the structure of the interface rather than the final result.
As stated in the literature
A mobile-agent benchmark whose verifier checks UI-tree structure rather than final device state; excluded from the terminal-agent study because that grading style doesn't fit the screen-free paradigm.
Why it matters: How a benchmark grades success determines which kinds of agents it can fairly test, and structure-based grading excludes approaches that never read the screen.
For example, instead of checking whether a reminder was actually saved, this benchmark checks whether the agent navigated the expected sequence of menus.
Heard on the show
“One benchmark, AndroidLab, was excluded because its verifier checks the interface structure instead, which doesn't fit the terminal paradigm.”Episode 157 — When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed