MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

2025-07-25

AI Summary

This paper addresses key challenges in evaluating cross-platform (Windows/macOS/Linux/iOS/Android/Web) GUI automation agents—namely, difficulty in performance assessment, low task efficiency, and excessive redundant actions. We propose the first hierarchical evaluation framework, spanning four capability levels: interface understanding, element localization, task automation, and collaborative reasoning. We introduce the Efficiency-Quality Area (EQA) metric to quantify online execution efficacy. Empirical analysis identifies precise visual localization as the primary bottleneck for task success and validates that modular architecture and early-stopping strategies are critical for efficiency gains. By integrating accurate localization, long-context memory, large action spaces, and long-horizon reasoning, the framework enables cross-platform generalization and joint modeling. Experiments demonstrate that coupling strong planning capabilities with a dedicated localization module significantly reduces redundancy and improves success rates—establishing a systematic, empirically grounded evaluation paradigm for scalable, high-performance GUI agents.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Intelligent Robots: Localization, Mapping, and Navigation
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.

Problem

• Evaluating GUI automation agents across multiple platforms
• Assessing GUI agent efficiency with EQA metric
• Improving task success via precise visual grounding

Innovation

• Hierarchical multi-platform GUI evaluation framework
• Efficiency-Quality Area metric for automation assessment
• Modular frameworks with precise grounding integration