DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

2026-07-15

AI Summary

This work addresses the challenge that existing agents struggle to collaboratively accomplish end-to-end tasks across heterogeneous devices—such as smartphones, desktops, and IoT systems—and lack a benchmark to effectively evaluate their capabilities in information acquisition and integration. To bridge this gap, we propose the first evaluation framework for cross-device collaborative task execution grounded in real user needs, introducing a large-scale executable benchmark comprising 6,140 tasks spanning three device environments. The framework employs natural language task specifications, rule-based validators, and a multi-stage quality control pipeline to enable automated assessment and diagnostic analysis. Experimental results reveal that even state-of-the-art LLM-based agents achieve a maximum success rate of only 12.5%, exposing critical limitations including device role confusion, insufficient information gathering, and incomplete task execution, thereby establishing a foundational benchmark for multi-device agent evaluation.

Abstract

LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. However, real-world user goals often span multiple devices: information may come from a phone, be processed on a desktop, and the result may need to appear on another device. Most existing benchmarks center on a single dominant execution environment, making it difficult to evaluate whether agents can acquire and integrate information across heterogeneous devices and complete end-to-end tasks with cross-device dependencies. We introduce DevicesWorld, a large-scale executable benchmark for cross-device collaborative operation. DevicesWorld contains 6,140 tasks and integrates three classes of device environments -- mobile, desktop, and IoT -- into a unified cross-device interaction and evaluation framework. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure. A multi-stage construction and quality-control pipeline keeps tasks close to realistic user needs while allowing final outcomes to be automatically verified from device states and generated files. We evaluate five frontier LLM-agent systems on a fixed evaluation set. All methods achieve low success rates, with the best reaching only 12.5%. Among failed runs, about 28.7% satisfy at least one scoring condition yet still fail the full task. Trajectories show that agents become stuck acquiring information or manipulating interfaces, confuse source and output devices, or terminate before all conditions are jointly satisfied. DevicesWorld turns cross-device collaborative operation into an executable, reproducible, and diagnostically useful evaluation problem for research on reliable cross-device agents.

Problem

• cross-device agents
• heterogeneous environments
• executable benchmark
• multi-device collaboration
• LLM-based agents

Innovation

• cross-device agents
• heterogeneous environments
• executable benchmark
• LLM-based agents
• task automation