MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization

2026-06-18

AI Summary

This work addresses the limitations of existing multimodal large language model (MLLM)-based mobile GUI agents, which rely heavily on manual annotations and lack a unified framework integrating exploration, curriculum learning, and policy optimization. The authors propose MobileForge, the first end-to-end adaptation framework that operates without human-labeled data. It leverages MobileGym to automatically generate tasks and interaction trajectories from real-world applications and introduces a Hierarchical Feedback-guided Policy Optimization (HiFPO) mechanism that fuses outcome-level, step-level, and corrective feedback signals to enable multi-granularity, context-aware policy updates. Evaluated on AndroidWorld using Qwen3-VL-8B, MobileForge achieves a 67.2% Pass@3 score; fine-tuning yields ForgeOwl-8B, which further improves performance to 77.6%. Notably, it attains a 41.0% success rate on out-of-domain GUI-only tasks, establishing a new state-of-the-art among open-source mobile GUI agents.

Abstract

MLLM-based mobile GUI agents have made substantial progress in UI understanding and action execution, but adapting them to real target apps remains costly because mobile apps are numerous, frequently updated, and hard to cover with human-written tasks, demonstrations, or reward labels. Existing annotation-free GUI learning reduces manual supervision, yet lacks a unified substrate connecting target-app exploration, curriculum mining, rollout execution, and feedback, while policy optimization often relies on isolated rollouts and coarse rewards that are hard to convert into reliable improvement signals. We present MobileForge, an annotation-free adaptation system for mobile GUI agents. MobileForge consists of MobileGym, which grounds task generation and rollout evaluation in real mobile app interaction, and Hierarchical Feedback-Guided Policy Optimization (HiFPO), which turns trajectory outcomes, step-level process feedback, and corrective hints into hint-contextualized step-level GRPO updates. Using only automatically generated annotation-free adaptation data, MobileForge adapts Qwen3-VL-8B to 67.2% Pass@3 on AndroidWorld, close to the closed-data GUI-specialized GUI-Owl-1.5-8B base model at 69.0%. The MobileForge-adapted ForgeOwl-8B further reaches 77.6% Pass@3 on AndroidWorld and 41.0% success on the out-of-domain MobileWorld GUI-only split, establishing the strongest open-data mobile GUI agent in our evaluation. Code, data, and trained models will be released at https://mobile-forge.github.io/.

Problem

• annotation-free adaptation
• mobile GUI agents
• policy optimization
• real-world app adaptation
• reward labeling

Innovation

• annotation-free adaptation
• mobile GUI agents
• hierarchical feedback
• policy optimization
• trajectory evaluation