InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization

2025-08-07

AI Summary

Multimodal large language models (MLLMs) exhibit weak semantic alignment and low exploration efficiency in GUI navigation, particularly struggling with complex semantic–spatial correlations. Method: This paper proposes the Adaptive Exploration Policy Optimization (AEPO) framework, featuring a multi-answer generation mechanism to broaden semantic coverage and an Efficiency-driven Adaptive Exploration Reward (AER) function—grounded in theoretical principles—to enable verifiable, reward-guided reinforcement learning. AEPO integrates MLLMs, reinforcement learning, and a formally verifiable reward mechanism to jointly optimize semantic fidelity and spatial localization accuracy. Contribution/Results: The framework trains InfiGUI-G1-3B/7B models that achieve state-of-the-art performance on multiple GUI localization benchmarks, outperforming prior baselines by up to 9.0% absolute accuracy gain while preserving semantic consistency.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The emergence of Multimodal Large Language Models (MLLMs) has propelled the development of autonomous agents that operate on Graphical User Interfaces (GUIs) using pure visual input. A fundamental challenge is robustly grounding natural language instructions. This requires a precise spatial alignment, which accurately locates the coordinates of each element, and, more critically, a correct semantic alignment, which matches the instructions to the functionally appropriate UI element. Although Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be effective at improving spatial alignment for these MLLMs, we find that inefficient exploration bottlenecks semantic alignment, which prevent models from learning difficult semantic associations. To address this exploration problem, we present Adaptive Exploration Policy Optimization (AEPO), a new policy optimization framework. AEPO employs a multi-answer generation strategy to enforce broader exploration, which is then guided by a theoretically grounded Adaptive Exploration Reward (AER) function derived from first principles of efficiency eta=U/C. Our AEPO-trained models, InfiGUI-G1-3B and InfiGUI-G1-7B, establish new state-of-the-art results across multiple challenging GUI grounding benchmarks, achieving significant relative improvements of up to 9.0% against the naive RLVR baseline on benchmarks designed to test generalization and semantic understanding. Resources are available at https://github.com/InfiXAI/InfiGUI-G1.

Problem

• Improving spatial alignment for GUI grounding
• Enhancing semantic alignment in MLLMs
• Addressing inefficient exploration in RLVR

Innovation

• Adaptive Exploration Policy Optimization (AEPO) framework
• Multi-answer generation for broader exploration
• Theoretically grounded Adaptive Exploration Reward (AER) function