Xiaomi-GUI-0 Technical Report

2026-06-30

AI Summary

This work addresses the limited generalization of existing GUI agents—trained in simulation—to real-world mobile applications, where interface variations, interaction discrepancies, and anomalous states pose significant challenges. The authors propose a native multimodal GUI agent designed for real mobile environments, supported by a hybrid closed-loop infrastructure that prioritizes physical devices with sandbox augmentation, enabling data collection, training, and evaluation under deployment-realistic distributions. By integrating multisource data encompassing high-frequency tasks, long-tail intents, and reflective memory, and introducing an error-driven data flywheel that converts failure trajectories into corrective actions and recovery demonstrations, the approach employs a three-stage progressive training pipeline: supervised fine-tuning, step-level reinforcement learning, and full-agent reinforcement learning, all grounded in vision-language models for end-to-end control. The method achieves success rates of 72.0% on RealMobile and 78.9% on AndroidWorld, substantially enhancing robustness and anomaly-handling capabilities in real-world scenarios.

Abstract

Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.

Problem

• GUI agents
• real-world deployment
• execution stability
• abnormal-state distribution
• benchmark-reality gap

Innovation

• real-device closed loop
• error-driven data flywheel
• multimodal GUI agent
• hybrid infrastructure
• agentic reinforcement learning