MobileWorldBench: Towards Semantic World Modeling For Mobile Agents

2025-12-15

AI Summary

Existing pixel-based world models struggle to accurately capture complex visual state transitions in mobile GUI environments. To address this, we propose a semantic world modeling paradigm that represents GUI states and their transitions using natural language descriptions instead of raw pixels. Methodologically, we introduce the first semantic modeling framework for mobile GUI agents, comprising (1) multimodal large model (VLM)-driven semantic state prediction, (2) alignment modeling of GUI action–feedback pairs, and (3) a collaborative integration mechanism between the VLM and the planning module. We further release MobileWorldBench—the first dedicated evaluation benchmark for mobile GUI world modeling—and MobileWorld, a large-scale dataset containing 1.4 million samples. Experiments demonstrate substantial improvements in task success rates for mobile agents, empirically validating the effectiveness and practicality of semantic world models in real-world GUI environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

World models have shown great utility in improving the task performance of embodied agents. While prior work largely focuses on pixel-space world models, these approaches face practical limitations in GUI settings, where predicting complex visual elements in future states is often difficult. In this work, we explore an alternative formulation of world modeling for GUI agents, where state transitions are described in natural language rather than predicting raw pixels. First, we introduce MobileWorldBench, a benchmark that evaluates the ability of vision-language models (VLMs) to function as world models for mobile GUI agents. Second, we release MobileWorld, a large-scale dataset consisting of 1.4M samples, that significantly improves the world modeling capabilities of VLMs. Finally, we propose a novel framework that integrates VLM world models into the planning framework of mobile agents, demonstrating that semantic world models can directly benefit mobile agents by improving task success rates. The code and dataset is available at https://github.com/jacklishufan/MobileWorld

Problem

• Develops a semantic world model for mobile GUI agents using natural language.
• Introduces a benchmark and dataset to evaluate and enhance vision-language models.
• Proposes a framework integrating semantic world models to improve agent task success.

Innovation

• Using natural language for GUI state transitions
• Introducing a benchmark for VLM world modeling
• Integrating semantic world models into agent planning