Problem
Research questions and friction points this paper is trying to address.
phone agents
mobile workflows
mixed-action
verifiable execution
GUI control
PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions
🤖 AI Summary
This work addresses the limitations of existing mobile agents, which are largely confined to GUI interactions and struggle with real-world tasks requiring integrated use of graphical interfaces, command-line tools, and external host-side utilities, while also lacking verifiable execution mechanisms. To overcome these challenges, the paper introduces the first mobile agent execution paradigm that supports hybrid invocation across GUI, CLI, and host-based tools. By leveraging deterministic action routing, constrained GUI delegation, and auditable execution traces, the framework shifts the task-completion criterion from predicting interface states to achieving observable side effects. Evaluated on the PhoneHarness Bench, the proposed approach achieves a 75.0% pass rate, outperforming the strongest baseline by 12.9 percentage points, thereby demonstrating the critical importance of hybrid action spaces and verifiable execution for effective mobile automation.
📝 Abstract
Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
Innovation
Methods, ideas, or system contributions that make the work stand out.
mixed-action agents
mobile workflow automation
verifiable execution
GUI-CLI-tool integration
action routing
🔎 Similar Papers
No similar papers found.
