Training Open Models for Agentic Phone Use

2026-06-22

AI Summary

Training open models in real smartphone environments faces significant challenges, including irreproducible settings, complex states, and numerous side effects. This work proposes PhoneBuddy, a novel training framework that synergistically combines real-device execution with PhoneWorld—a high-fidelity, resettable simulated environment reconstructed from real GUI structures. By integrating supervised fine-tuning with hybrid reinforcement learning that leverages both real and simulated environments, PhoneBuddy substantially enhances task execution capabilities. Experimental results demonstrate that on 150 real-world mobile tasks, the success rate improves from 36.67% under supervised fine-tuning to 45.33% with hybrid RL. Furthermore, on the AndroidWorld benchmark, performance increases from 60.3% to 83.2%, validating the effectiveness and complementary advantages of jointly training in real and simulated environments.

Abstract

Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL. On AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.

Problem

• agentic phone use
• real-app environment
• mock environment
• open models
• cross-app workflows

Innovation

• agentic phone use
• mock-app environment
• PhoneWorld
• mixed reinforcement learning
• supervised fine-tuning