HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

2026-07-16

AI Summary

This work addresses the intertwined challenges faced by mobile GUI agents—namely, complex interface perception, scalable acquisition of high-quality interaction data, and error accumulation in long-horizon decision-making—by proposing an agent framework built upon the vision-native large model Hy3.0-VL-A3B, which supports arbitrary-resolution inputs and a 32K-token context window. The approach introduces a novel “data-environment co-scaling” paradigm, featuring a GUI perception flywheel, the PhoneWorld Mock App Factory simulation environment, and a structured plan-reflection mechanism augmented with infinite-loop detection. Leveraging synthetic UI generation, structured tutorial video parsing, million-scale action collection, and staged training, the system achieves efficient and robust long-range interactions across 34 simulated applications, over 34,000 tasks, and more than 2,000 real or sandboxed devices, substantially improving task success rates and execution reliability.

Abstract

As large multimodal models move from understanding content to operating on digital environments, mobile GUI has emerged as a challenging and consequential testbed for digital embodied intelligence. Mobile agents operate under three coupled constraints: precise perception of complex interfaces, scalable acquisition of high-quality interaction data, and robust long-horizon decision making under compounding execution errors. This report presents HyMobileAgent, a mobile GUI agent built on Hy3.0-VL-A3B, a vision-native foundation model featuring native any-resolution input, an A3B-scale deployment budget, and a 32K context window to model extended interaction histories. Rather than relying solely on model scaling, we develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction. Our framework integrates a GUI perception flywheel combining mock-interface synthesis, rejection sampling, and icon-specific augmentation; a knowledge pipeline that transforms tutorial videos into structured interaction data; a million-scale action data pipeline deployed across more than 2000 sandbox and real-device instances with automated failure attribution; the PhoneWorld Mock App Factory, providing a resettable training environment with 34 mock applications and over 34000 tasks; and a structured Planning-and-Reflection mechanism with explicit dead-loop detection for reliable long-horizon execution. We also introduce a progressive training recipe consisting of mid-training, supervised fine-tuning, and reinforcement learning with task-specific reward designs.

Problem

• mobile GUI agents
• digital embodied intelligence
• long-horizon decision making
• interaction data scaling
• GUI perception

Innovation

• data-environment co-scaling
• GUI perception flywheel
• structured Planning-and-Reflection
• mock app factory
• progressive training recipe