Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

2025-04-01

AI Summary

To address three key challenges in GUI agents—imprecise element grounding, difficulty in long-horizon task planning, and limited cognitive capabilities of general-purpose models—this paper proposes a novel computer-use agent specifically designed for GUI interaction. Methodologically: (1) We introduce Mixture-of-Grounding, a hybrid localization technique that enhances fine-grained perception of GUI elements; (2) we design Proactive Hierarchical Planning to enable multi-scale, dynamic task decomposition and reconstruction; and (3) we establish a collaborative architecture integrating general-purpose foundation models with specialized modules for visual grounding, hierarchical reasoning, real-time observational feedback, and cross-platform adaptation. Evaluated on OSWorld (15/50-step), WindowsAgentArena, and AndroidWorld, our agent achieves comprehensive performance gains over state-of-the-art baselines—improving success rates by 18.9%–52.8%—and demonstrates significantly enhanced cross-OS generalization capability.

Technology Category

• Multiagent Systems: Multiagent Planning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Intelligent User Interfaces

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibilities across various generalist and specialist models. We propose a novel Mixture-of-Grounding technique to achieve precise GUI localization and introduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.

Problem

• Improves GUI element grounding precision for agents
• Enhances long-horizon task planning capabilities
• Optimizes performance using generalist-specialist model composition

Innovation

• Mixture-of-Grounding for precise GUI localization
• Proactive Hierarchical Planning for dynamic task refinement
• Compositional generalist-specialist framework for diverse tasks