ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks

2026-06-19

AI Summary

Current evaluations of computer-using agents are largely confined to atomic desktop tasks, failing to capture the ability to maintain state across goals in realistic, long-horizon workflows. This work proposes ChainWorld—the first benchmark designed for evaluating long-horizon desktop automation—by composing OSWorld atomic tasks into 347 task chains of length 2–4 via directional compatibility search, and introducing both single-turn and multi-turn evaluation protocols. Experiments reveal that leading agents achieve only a 31% success rate on the longest chains. Multi-turn interaction substantially improves performance across three model families, while failure mode analysis shows that single-turn failures primarily stem from insufficient output precision, whereas multi-turn settings expose deficiencies in conversation management.

Abstract

Computer use agents are evaluated almost exclusively on atomic desktop tasks, but realistic desktop work requires sustaining state across multiple objectives. We study this gap with ChainWorld, which composes atomic OSWorld tasks into long horizon desktop workloads through directional compatibility search while preserving the source evaluators. The resulting workload contains 347 chains of length two to four and compares two renderings of the same task sequence. In single turn evaluation, all tasks are presented together in one prompt. In multi turn evaluation, tasks are revealed one at a time. Across four current computer use agents, maximum chain completion is 31%. Multi turn evaluation improves completion for three models, but both protocols remain challenging. The two protocols also expose different failure profiles. Single turn failures concentrate on artifact precision, while multi turn failures more often reflect session management problems such as fragmented progress and later turn disengagement.

Problem

• long-horizon tasks
• desktop workloads
• computer use agents
• task chaining
• stateful interaction

Innovation

• ChainWorld
• long-horizon tasks
• directional compatibility search
• multi-turn evaluation
• desktop automation