UI-TARS: Pioneering Automated GUI Interaction with Native Agents

2025-01-21

AI Summary

This paper addresses core challenges in GUI agents—limited cross-platform generalization, imprecise action grounding, and weak long-horizon planning—by proposing UI-TARS, an end-to-end native GUI agent. UI-TARS takes raw screen screenshots as input and directly outputs pixel-level keyboard/mouse actions, eliminating reliance on large language model wrappers or manual prompt engineering. Methodologically, it integrates enhanced visual perception, unified cross-platform action modeling, System-2 multi-step reasoning (including task decomposition, milestone identification, and reflective refinement), and an iterative training paradigm grounded in online trajectory reflection. High-quality, cross-platform GUI trajectories are automatically collected, filtered, and optimized via a virtual machine cluster; a standardized action space and large-scale GUI screenshot dataset are constructed. UI-TARS achieves state-of-the-art performance across 10+ benchmarks—including OSWorld (24.6/50 steps) and AndroidWorld (46.6)—outperforming GPT-4o, Claude, and other SOTA methods.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions (e.g., keyboard and mouse operations). Unlike prevailing agent frameworks that depend on heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts and workflows, UI-TARS is an end-to-end model that outperforms these sophisticated frameworks. Experiments demonstrate its superior performance: UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating perception, grounding, and GUI task execution. Notably, in the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld, UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of GUI screenshots for context-aware understanding of UI elements and precise captioning; (2) Unified Action Modeling, which standardizes actions into a unified space across platforms and achieves precise grounding and interaction through large-scale action traces; (3) System-2 Reasoning, which incorporates deliberate reasoning into multi-step decision making, involving multiple reasoning patterns such as task decomposition, reflection thinking, milestone recognition, etc. (4) Iterative Training with Reflective Online Traces, which addresses the data bottleneck by automatically collecting, filtering, and reflectively refining new interaction traces on hundreds of virtual machines. Through iterative training and reflection tuning, UI-TARS continuously learns from its mistakes and adapts to unforeseen situations with minimal human intervention. We also analyze the evolution path of GUI agents to guide the further development of this domain.

Problem

• Automated GUI Operation
• Cross-platform Capability
• Self-learning Optimization

Innovation

• Autonomous Screen Recognition
• Cross-Platform Precision
• Human-like Decision Making