AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

2025-06-02

AI Summary

Existing GUI agents for mobile platforms suffer from noisy training data, insufficient semantic diversity, weak cross-lingual generalization, and challenges in on-device deployment—particularly within the Chinese mobile ecosystem. To address these issues, this paper introduces the first 8B-parameter on-device GUI agent tailored for Chinese mobile environments. Methodologically: (1) we propose grounding-aware pretraining to enhance UI understanding and action grounding; (2) we construct a high-quality bilingual (Chinese–English) trajectory dataset and apply multilingual supervised fine-tuning; (3) we integrate GRPO-based reinforcement fine-tuning with a compact discrete action space to improve cross-app generalization and robust decision-making. Our agent achieves state-of-the-art performance on five public benchmarks and our newly established Chinese CAGUI benchmark (Type-Match: 96.9%, Exact-Match: 91.3%), while enabling low-latency on-device inference. The code, model, and evaluation datasets are fully open-sourced.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability. However, practical deployment of such agents remains constrained by several key challenges. Existing training data is often noisy and lack semantic diversity, which hinders the learning of precise grounding and planning. Models trained purely by imitation tend to overfit to seen interface patterns and fail to generalize in unfamiliar scenarios. Moreover, most prior work focuses on English interfaces while overlooks the growing diversity of non-English applications such as those in the Chinese mobile ecosystem. In this work, we present AgentCPM-GUI, an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction. Our training pipeline includes grounding-aware pre-training to enhance perception, supervised fine-tuning on high-quality Chinese and English trajectories to imitate human-like actions, and reinforcement fine-tuning with GRPO to improve reasoning capability. We also introduce a compact action space that reduces output length and supports low-latency execution on mobile devices. AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks and a new Chinese GUI benchmark called CAGUI, reaching $96.9%$ Type-Match and $91.3%$ Exact-Match. To facilitate reproducibility and further research, we publicly release all code, model checkpoint, and evaluation data.

Problem

• Noisy and semantically limited training data hinders precise GUI interaction learning
• Models overfit to seen interfaces and fail in unfamiliar scenarios
• Prior work neglects non-English interfaces like Chinese mobile ecosystems

Innovation

• Reinforcement fine-tuning with GRPO
• Grounding-aware pre-training for perception
• Compact action space for mobile efficiency