Definition
Plain language
A benchmark that tests AI agents on real Android phone apps.
As stated in the literature
A mobile-device agent benchmark covering roughly twenty apps and over a hundred tasks with state-based verification of cross-app workflows.
Why it matters: Mobile interfaces are different enough from desktop browsers that they need their own benchmark, and this is a leading one for evaluating phone-using agents.
For example, an AndroidWorld task might ask an agent to find a contact in one app, copy the phone number, and paste it into a text message in another.
Heard on the show
“AndroidWorld is twenty apps, a hundred-sixteen tasks.”Episode 017 — When the Agent Grades Its Own Homework: A Brutal New Benchmark for AI Workers