WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments

2026-04-30

AI Summary

This work addresses the limitation of existing GUI agent benchmarks, which are often confined to single-application tasks and thus inadequate for evaluating complex, cross-application interactions required in professional workflows. To bridge this gap, we introduce WindowsWorld—the first benchmark driven by real-world occupational workflows—comprising 181 human-refined tasks spanning 16 professions and four difficulty levels, with 78% involving multiple applications. Implemented in a simulated desktop environment, WindowsWorld encompasses 17 commonly used applications and supports fine-grained subgoal tracking and intermediate state validation. Empirical evaluation reveals that state-of-the-art agents achieve less than 21% success on multi-application tasks, with particularly poor performance on conditional reasoning tasks involving three or more applications, underscoring the benchmark’s critical value in assessing professional-grade human-computer interaction capabilities.

Abstract

While GUI agents have shown impressive capabilities in common computer-use tasks such as OSWorld, current benchmarks mainly focus on isolated and single-application tasks. This overlooks a critical real-world requirement of coordinating across multiple applications to accomplish complex profession-specific workflows. To bridge this gap, we present a computer-use benchmark in cross-application workflows, named WindowsWorld, designed to systematically assess GUI Agents on complex multi-step tasks that mirror real-world professional activities. Our methodology uses a multi-agent framework steered by 16 occupations to generate four difficulty-level tasks with intermediate inspection, which are then refined by human review and executed in a simulated environment. The resulting benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. Experimental results of leading large models and agents show that: 1) All computer-use agents perform poorly on multi-application tasks (< 21% success rate), far below the performance of simple single-app tasks; 2) They largely fail at tasks requiring conditional judgment and reasoning across $\geq$ 3 applications, stalling at early sub-goals; 3) Low execution efficiency, where tasks often fail despite far exceeding human step limits. Code, benchmark data, and evaluation resources are available at github.com/HITsz-TMG/WindowsWorld.

Problem

• GUI agents
• cross-application workflows
• professional tasks
• benchmarking
• multi-step tasks

Innovation

• cross-application workflows
• GUI agents
• process-centric benchmark
• multi-agent task generation
• professional desktop automation