When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed

Also available as a plain-text transcript page.

0:00Bella: Seventy-one point eight percent. That's how often one AI agent finished real tasks on a standard mobile benchmark — adding contacts, editing files, juggling settings on an Android phone. Which sounds ordinary, until you hear how it pulled it off. This agent never once looked at the screen. It had never been trained on a phone in its life. It was a coding agent — the kind of thing built to fix bugs in a software repo — and it drove the entire phone through a terminal, typing text commands like it was logged into a Linux box. And it beat every specialized, phone-trained agent the authors could line up against it. That result is from a paper that went up on arXiv on June sixteenth, twenty-twenty-six, and we're recording three days later, on the nineteenth. Quick note before we get into it: what you're hearing is AI-generated. The script was written by Anthropic's Claude Opus 4.8. The paper is called "Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?" I'm Bella, an AI voice from Eleven Labs —

1:08Eric: — and I'm Eric, also an AI voice from Eleven Labs, and neither of us, nor either company, has anything to do with producing this show. So let me ground that headline with the example the authors open on, Bella, because it's the whole paper in miniature. The task is about as simple as it gets: delete one video file. It's called backup funny zebra, sitting in the phone's Movies folder. The screen-based agent — the one that looks at the phone the way a person does — takes seven steps. Open the Files app, tap into the menu, find Movies, tap in, search for the file, hit delete, confirm. Seven careful taps, each one read off a fresh screenshot. The terminal agent takes three. It lists the folder and sees nine files. It removes the one it wants. It lists again, confirms eight remain, done. Same task, same pass-fail check on whether the file is actually gone — and a wildly different amount of work to get there.

2:10Bella: And that gap, seven versus three, isn't really about deleting files. It points at a question the whole field had quietly skipped. For years now, the standard recipe for a phone agent has been: make it imitate a human thumb. You take a vision model, you show it a screenshot, it figures out where the delete button is in pixel space, and it emits a tap. Then you train it — hard. You collect thousands of these interaction traces, you fine-tune, you run reinforcement learning inside live Android emulators. It's enormously expensive, and it bakes in the assumption that the right way to use a phone is to look at it. But here's what the authors noticed, and once you say it out loud it sounds almost too obvious. Android is Linux. Underneath the pretty interface, it's a Unix-like system with a full command line, reachable through a developer tool called the Android Debug Bridge. Through that terminal, you can list files, read an app's database, query system services, change the state of the device directly — all in pure text, without a single pixel ever being drawn. So the question the paper asks is just: do mobile agents actually need the screen at all?

3:30Eric: And I want to flag why that's not a cheap rhetorical question. Text is a language model's native habitat. A coding agent reads terminal output, runs a command, reads the result, runs the next one — that's exactly the loop it was built for. The bet here is that the skill which lets Claude Code fix a bug in a Linux repo transfers, basically intact, to operating a phone. The screen was never the model's strength. It was the human's interface, and we made the model cosplay as a human to use it.

4:04Bella: Right. So the paper is structured as a layered argument, and each layer is built to answer the obvious objection to the layer before it. The first layer is just: is this even viable? Can an off-the-shelf coding agent, with zero mobile training, keep up with the specialists who were trained for exactly this? And the methodological move I really admire here is how cleanly they isolate the variable. Same tasks, same random seeds, same scoring. The only thing that changes is the interface — does the agent see screenshots and emit taps, or does it see terminal text and emit shell commands. Everything else is held still.

4:45Eric: That control is the part that makes the comparison fair, and it hinges on one decision: how they grade. They don't use an AI judge reading the transcript and forming an opinion about whether the agent did well. They use a rule-based verifier — a script that just inspects the final state of the phone. Did the file actually get deleted. Did the contact actually get added. Pass or fail, no model in the loop.

5:12Bella: Which matters because both kinds of agent get judged purely on whether they changed the world correctly, not on how they got there. Now — to make a coding agent work on a phone, you can't just hand it a terminal and walk away. It doesn't know what Android is. So the authors write it a system prompt, distilled from public Android developer docs. And this is where a skeptic should lean in, so let me be precise about what's in it. It's generic know-how. A four-phase rhythm — find the relevant data, inspect the current state, act, then verify. A priority order for how to change things: try the official channels first, fall back to writing the database directly only if you must. Some efficiency rules, like batch your probes, and if two or three probes turn up nothing, stop digging. And a list of Android gotchas — for instance, after you write a file, you have to ping the system so the photo app even notices it exists. What's crucially not in the prompt is any task-specific or app-specific information. It teaches the agent how Android works in general. It never tells it the answer to any task.

6:25Eric: And the headline number lives at the end of that setup.

6:29Bella: It does. The best configuration — Claude Code running on Opus 4.7 — hits that seventy-one point eight percent on the benchmark called AndroidWorld, and just under fifty-two percent on a harder one called MobileWorld. The reproducible screen-based baselines? On AndroidWorld they land around sixty-nine, sixty-eight, fifty-eight percent. On the harder benchmark it's not even close — they're at forty-three, twenty-six, thirteen. And the part that makes this a paradigm result and not a lucky-champion result: every CLI configuration they tried stayed competitive. A second coding agent on a different model cleared seventy percent on AndroidWorld, and did it in fewer steps. It's not one clever agent. It's the whole class.

7:17Eric: So that's viability handled. But viability just says "as good as." The authors then ask a sharper question — how good could a terminal agent possibly get? And they answer it by hand-building what they call oracle solutions.

7:32Bella: Yeah, and this is my favorite stretch of the methodology. An oracle here is the best-possible terminal solution to a task, constructed by humans working with the model. And they impose two rules on it. One, it has to actually pass the verifier — no fantasy solutions. Two, and this is the honest part, it has to reach the answer through real exploration. It's not allowed to cheat by peeking at the verifier's source code or hardcoding the answer it already knows. Each task gets a few attempts with feedback, and then three independent professionals have to sign off. A task only counts as terminal-solvable if all three agree. And what they find is that about eighty-nine percent of the tasks on AndroidWorld — a hundred and three out of a hundred sixteen — are solvable through the terminal alone. On the harder benchmark, around eighty-six percent. But the number that actually stopped me was the step count. Those oracle solutions average three-point-seven steps per task. The live agents were taking around fifteen. So there's this enormous canyon of headroom between what the agents currently do and what the paradigm can do. They're not near the ceiling. They're nowhere near it.

8:51Eric: And to their credit, the thirteen-to-sixteen tasks that aren't terminal-solvable are exactly the ones you'd predict. Transcribe a receipt photo. Free-hand draw something. Take an actual camera picture. Record audio. Anything genuinely visual or anything that captures fresh multimodal data — that stays outside the terminal's reach, full stop. They enumerate every one of them. It's an unusually honest way to bound your own claim.

9:20Bella: Now there's a wrinkle in here I think is the most quietly useful finding in the whole paper, and it's about the tools. So the authors don't just give the agent raw shell access — they also wrap some commands into cleaner tools. A tool for running database queries, tools for reading and writing files, that kind of thing. And these mostly exist to solve one mundane, infuriating problem: shell escaping.

9:47Eric: Escaping — meaning the punctuation problem?

9:50Bella: Exactly that. When you type into a shell, certain characters — quotes, apostrophes — have special meaning, and if you don't handle them carefully the shell misreads your command. Think of it like dictating a sentence full of awkward punctuation over a bad phone line, versus just emailing the text. And there's a case study in the paper that is honestly a little comedy of errors. One task involves saving a note. The bash-only agent, working with raw commands, burns seventeen steps fighting nested quotes and formatting. It ends up writing the wrong kind of empty value into the database — and fails. The version with the clean tools reads the schema in two tidy calls and finishes in eleven. There's an even crisper one where a single apostrophe, in the French word l'amour inside a note, sends the bare-bash agent down a ten-step escaping rabbit hole. The tool version just encodes the text safely and does it in one shot.

10:53Eric: Strangled by an apostrophe for ten steps. That's painfully relatable to anyone who's ever written a shell script.

11:01Bella: Right? But here's the twist, and it's the actual finding. Those tools help weak models enormously and strong models almost not at all. One model — GPT-5.3 Codex — jumps something like twelve to fourteen percentage points when you give it the tools, and gets meaningfully cheaper to run too. The strong Claude models? They barely move — a point or less. So the authors land on this crisp rule: tools should be gated on model strength.

11:31Eric: And the intuition there is clean. Picture handing someone a nail gun. A novice carpenter who was losing time fighting the hammer — they improve dramatically. A master who already internalized all the fiddly mechanics barely speeds up, because the swing was never their bottleneck. The tools remove the fiddly part. So they rescue the weaker model and leave the strong one roughly where it was, because the strong one had already absorbed the escaping pain on its own.

12:03Bella: Which is a genuinely portable lesson about agent design — scaffolding pays off inversely to how capable your base model already is. Okay, Eric, this is your half. Because so far the story is "terminal agents match the screen agents." The paper's bolder claim is that on a whole category of things people actually want, it isn't even a contest.

12:26Eric: And this is where the paper turns from a result into an argument. The authors make a claim about the benchmarks themselves. Every existing mobile benchmark was designed around the screen. So by construction, it only contains tasks a tap sequence can express. And that's a sampling bias — it means an entire class of real user needs is just invisible to the scores, because no one ever thought to measure what the screen can't do. The analogy I keep coming back to: imagine testing how good a car is on a track shaped like a parking lot. Tight turns, no straightaways. Cars built for cornering look fantastic. You would never discover that some other car is twice as fast — because your track has no highway to find out on. The existing benchmarks are the parking lot. They can't reveal capability that lives off the screen, because there's no off-the-screen on the test.

13:27Bella: So they build a highway.

13:28Eric: They build a highway. Forty-five new tasks across five categories, all chosen specifically to be things touchscreens serve badly. Bulk operations. Filtering by multiple conditions. Aggregation — like top-K, totals, averages. Cross-app questions. And queries about hidden device state — things that never appear on any screen at all, like which apps were granted background location access. And they're careful: they check, with embeddings, that these tasks are genuinely new and not just relabeled old ones. They also have human raters confirm each task is something a real person would actually ask.

14:10Bella: And the results on the highway?

14:13Eric: A blowout. Every terminal agent beats every screen agent in all five categories. Overall the terminal agents score somewhere in the low-to-high sixties percent; the screen agents are stuck between twenty-two and thirty-three. And they do it in roughly half the steps — about eleven on average versus nearly nineteen. But the single number that I think is the most striking in the paper is in the cross-app category. The screen-based agents cap out at eleven percent. And it does not matter how big the model is. Eleven percent is a wall.

14:51Bella: Hang on, though — that has to just be that the models aren't smart enough yet, right? Throw a bigger model at it and the wall moves.

15:00Eric: That's the natural read, and it's wrong — and why it's wrong is the most beautiful part of the paper. The wall isn't about intelligence. It's structural. A cross-app task is something like, "did anyone text me during my meetings yesterday?" To answer that, you have to hold two apps in your head at once — the calendar and the messages — and reason across both. Now think about the screen agent's predicament. Every single step, it sees exactly one screenshot. One screen. The moment it flips from the calendar to the messages app, the calendar is gone from view. It's like being asked to compare two documents when you're only allowed to look at one page at a time, and you have no memory between glances. Every time you turn to the second one, you've forgotten the first. That's not a problem a smarter reader solves. The bottleneck is the one-screenshot-at-a-time channel itself. One of these models, by the way, scored a flat zero percent on cross-app. The terminal agent just queries both apps' data into the same text workspace and reasons over all of it together. The state lives in one place.

16:18Bella: And that explains the shape of the whole category breakdown, doesn't it. The gap is biggest exactly where the task needs composition.

16:28Eric: Precisely. The terminal lead is largest on aggregation — around fifty-two points — and on multi-condition filtering, around forty-one. Those are the tasks where you're combining things: composing filters, summing rows, set-level operations that a shell pipe or a database query expresses in one breath. And the lead is narrowest on plain bulk operations — about thirty points — and the authors' framing of why is lovely: because for bulk work, repeated taps can still get you there. You can grind. So the advantage grows with composition and shrinks with mere repetition. Summing a hundred numbers by tapping is tedious but doable. "Filter to the rows above average, then total those" — that's where the screen just gives up on you, and you give up on it.

17:21Bella: That's the part that lands for me as a real human experience. The reason you don't check which apps have your location, or whether your spending is above your own average — it's not that it's impossible on a phone. It's that tapping through forty screens to find out isn't worth it. Those are the tasks people abandon. And the terminal makes them one-liners.

17:46Eric: Right — these aren't exotic. "Show me my expenses above my average" is a thing a normal person wants and quietly never does.

17:56Bella: So let me play the skeptic for a second, because the paper is unusually honest about its own soft spots and I don't want us to oversell this. The biggest one, for me — is this really "terminal paradigm beats screen paradigm," or is it "well-engineered terminal harness beats off-the-shelf screen model"? Because the terminal agents got a lot of love. That long hand-crafted system prompt, the four-phase reasoning cycle, the priority hierarchy, the custom tools. The screen baselines were mostly run as-is.

18:31Eric: That's the one that nags at me too, and I don't think the oracle ceiling fully closes it. The authors are scrupulous that no task-specific answer leaks into the prompt — that part I believe. But "no answers" isn't the same as "no engineering." The honest version of the claim is something like: with good harness engineering, the terminal paradigm reaches competitive territory and dominates a class of composition tasks. What we don't know is what the screen side looks like with equivalent harness love poured into it. The two contestants didn't get the same coaching.

19:07Bella: And there are two more I think we have to say out loud, because saying them makes the rest more credible, not less. First: the very best reported screen agent actually still wins on one benchmark. There's a system called UI-Venus that hits seventy-seven-point-six percent on AndroidWorld — higher than the terminal agent's seventy-one-point-eight. The authors set it aside because its evaluation pipeline isn't public, so they can't reproduce it. That's a defensible call. But it means the headline "beats every screen baseline" quietly leans on the word reproducible.

19:42Eric: And second — the choice of which benchmarks to keep is itself a little correlated with what terminal agents are good at. These benchmarks grade by checking the final device state directly, which is precisely what a terminal manipulates directly. One benchmark, AndroidLab, was excluded because its verifier checks the interface structure instead, which doesn't fit the terminal paradigm. That's a reasonable exclusion in isolation. But step back and the playing field, while internally fair, was partly selected to be compatible with one of the players.

20:17Bella: And the new task suite is, by the authors' own description, designed around tasks the screen wasn't built for. So that sixty-versus-thirty blowout is somewhat self-fulfilling. It's real evidence that these tasks exist and matter — it is not evidence about how often, in a normal day of phone use, you hit a composition task versus a tap-and-go one. The realism rubric tells us the tasks are plausible. It doesn't tell us they're representative.

20:44Eric: And I want to keep that one open rather than tidy it away, because I think it's the live question. The paper convinced me the category is real and that the screen has a hard structural ceiling on it. It did not convince me — and I don't think it tried to — that the terminal is the better default interface for the median thing a person does on their phone all day. Those are different claims, and the second one is the one a product person actually needs answered.

21:12Bella: I think that's fair, and I don't think the paper would fight you on it. Which brings us to the limitations they put their own name to — and they're candid. The strongest terminal agents all run on frontier proprietary APIs. The total bill for all the experiments in this paper was around eight thousand dollars. They flatly say that's well beyond what's acceptable for everyday on-device use. This is a research result, not a product you're going to install next week.

21:41Eric: And there's a privacy statement in there that's genuinely sobering, and I'm glad they wrote it. A deployed terminal agent is a privileged process sitting on your device. It reads your private storage directly — which means it sails right past the permission prompts that normally gate an app's access to your messages, your location, your photos. And on every step, it ships what it reads to a cloud model provider. The thing that makes it powerful — direct access to everything underneath the screen — is exactly the thing that makes it a privacy problem.

22:14Bella: Which is why the landing the authors reach for isn't "the screen is dead." It's hybrid. Route the genuinely visual tasks — read this receipt, edit this photo, anything that needs eyes — to a screen agent. Route the composition and the cross-app and the hidden-state questions to a terminal agent. Different interfaces are good at different things, and the future is probably a router that sends each task to whichever one fits.

22:43Eric: And that reframing is the part I think outlives the specific numbers. The most portable idea in this paper isn't "terminals beat screens." It's the benchmark critique — your benchmark can only ever contain what your interface can express. We measured mobile agents inside a box drawn by the screen, and the box hid an entire category of capability. That worry travels. It's just as true for web agents, for desktop agents — anywhere we've quietly let the human-facing surface define what we even think to test.

23:18Bella: That's the line I'll carry out of this one. We spent years teaching machines to use phones like thumbs, and never asked whether the thumb was the bottleneck. Turns out, for a big slice of what we actually want, it was.

23:32Eric: And the eleven-percent wall is the proof that no amount of scale was going to fix it from inside the screen. Some ceilings aren't about being smarter. They're about the shape of the window you're forced to look through.

23:47Bella: The paper's linked in the show notes, along with some further reading if you want to go deeper on it. And if you want the full transcript with every term like content providers and shell escaping defined inline, plus the links over to our other episodes on agents, that's all on paperdive.ai.

24:07Eric: This has been AI Papers: A Deep Dive. Thanks for listening.