GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection

2025-12-10

AI Summary

GUI automation tasks exhibit high diversity, leading to heterogeneous model capabilities and tight coupling with domain-specific expertise. Method: This paper proposes a multi-model collaborative intelligent agent framework that decouples capabilities through a joint inference architecture integrating a general-purpose large language model with multiple GUI-specialized models. It introduces a novel two-phase collaboration paradigm—“joint information reasoning” followed by “grouped reflection”: during reasoning, it fuses multi-source GUI state perception and action planning; when information is insufficient, it triggers instruction-driven grouped reflection to guide complementary, high-value information acquisition. Contribution/Results: Experiments demonstrate significant improvements in task completion rate and action accuracy across multiple GUI benchmarks. The framework exhibits superior robustness compared to single-model baselines and conventional ensemble methods.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Building AI systems for GUI automation task has attracted remarkable research efforts, where MLLMs are leveraged for processing user requirements and give operations. However, GUI automation includes a wide range of tasks, from document processing to online shopping, from CAD to video editing. Diversity between particular tasks requires MLLMs for GUI automation to have heterogeneous capabilities and master multidimensional expertise, raising problems on constructing such a model. To address such challenge, we propose GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection, a novel MLLM-based GUI automation agent framework designed for integrating knowledge and combining capabilities from heterogeneous models to build GUI automation agent systems with higher performance. Since different GUI-specific MLLMs are trained on different dataset and thus have different strengths, GAIR introduced a general-purpose MLLM for jointly processing the information from multiple GUI-specific models, further enhancing performance of the agent framework. The general-purpose MLLM also serves as decision maker, trying to execute a reasonable operation based on previously gathered information. When the general-purpose model thinks that there isn't sufficient information for a reasonable decision, GAIR would transit into group reflection status, where the general-purpose model would provide GUI-specific models with different instructions and hints based on their strengths and weaknesses, driving them to gather information with more significance and accuracy that can support deeper reasoning and decision. We evaluated the effectiveness and reliability of GAIR through extensive experiments on GUI benchmarks.

Problem

• Integrates heterogeneous models for diverse GUI automation tasks
• Enhances decision-making via joint reasoning and group reflection
• Addresses insufficient information in MLLM-based GUI automation systems

Innovation

• Integrates knowledge from heterogeneous GUI-specific MLLMs
• Uses a general-purpose MLLM for joint reasoning and decision-making
• Implements group reflection to gather more accurate information