Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?

2026-06-15

AI Summary

Existing mobile agents are constrained by the graphical user interface (GUI) paradigm, struggling to efficiently handle complex tasks such as batch operations and cross-application workflows. This work presents the first systematic exploration of command-line interfaces (CLIs) as an alternative interaction paradigm, enabling direct invocation of device services and data without requiring screen perception or touch-based actions. We introduce CLI-Advantage, a benchmark suite encompassing five representative scenarios challenging for GUI-based approaches, along with open-sourced evaluation infrastructure. Experimental results demonstrate that, without any mobile-specific fine-tuning, general-purpose code large language models—such as Claude Code—combined with Android CLI toolchains achieve success rates of 71.8% on AndroidWorld and 51.9% on MobileWorld, substantially outperforming current GUI-based baselines. Moreover, these CLI-driven agents accomplish tasks in an average of 10.7 steps, significantly fewer than the 18.6 steps required by GUI counterparts.

Abstract

Recent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information and emit screen interactions. However, mobile platforms also expose a command-line interface (CLI) that provides direct access to device services and data. We argue CLI deserves first-class consideration alongside GUI. We evaluate three coding agents (Claude Code, Terminus-2, mini-swe-agent) across four model APIs on AndroidWorld and MobileWorld without any mobile-specific post-training, comparing against three reproducible GUI baselines (GUI-Owl-1.5-32B, MAI-UI, Qwen3-VL-32B). Claude Code (Opus 4.7) reaches 71.8\% and 51.9\%, outperforming every reproducible GUI baseline (69.3/68.1/57.8\% on AndroidWorld; 43.2/26.3/13.3\% on MobileWorld), while every other CLI configuration remains competitive. To establish the paradigm's ceiling, we provide oracle CLI solutions that reach 88.8\% on AndroidWorld (103/116 tasks CLI-solvable) and 86.3\% on MobileWorld (101/117 tasks CLI-solvable), indicating substantial room for future improvement. To cover everyday user intents beyond the GUI scope, we introduce the \textbf{CLI-Advantage Task Suite}, comprising 45 templates across five categories: bulk operations, multi-condition filtering, aggregation, cross-app workflows, and hidden device state. Every CLI agent outperforms every GUI baseline in all five categories, with substantially fewer steps per task (10.7 vs.\ 18.6). To support future research on mobile CLI agents, we will open-source agent implementations, oracle solutions, the CLI-Advantage suite, and evaluation infrastructure.

Problem

• mobile agents
• command-line interface
• GUI paradigm
• CLI-Advantage
• user intents

Innovation

• mobile agents
• command-line interface (CLI)
• GUI paradigm
• CLI-Advantage Task Suite
• cross-app workflows