Definition
Plain language
A benchmark that tests phone-controlling AI agents on real physical devices instead of simulators.
As stated in the literature
A real-device mobile-agent benchmark from Xiaomi covering abnormal states (login walls, captchas, risk-control prompts) that emulators can't reproduce; used to expose the gap between emulator scores and real-world performance.
Why it matters: By testing on actual phones with messy real-world interruptions, it reveals whether an agent's polished simulator scores hold up where it actually has to work.
For example, RealMobile can confront an agent with a real captcha or a surprise login wall that a clean emulator would never throw at it.
Heard on the show
“But on RealMobile, their real-device benchmark, it hits seventy-two percent, against thirty-three for that comparable open model we opened with.”Episode 189 — Why Phone Agents Ace the Test and Crash on Your Actual Phone