JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

2026-09-09

AI Summary

该研究通过引入JarvisGUI，一种评估跨设备GUI代理性能的动态基准，解决了现有GUI代理在多平台协调工作流中的不足。

Abstract

Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.

Problem

• Cross-Device
• GUI Agents
• Heterogeneous Platforms
• State-Transfer Awareness
• Contextual Reasoning

Innovation

• Cross-Device Workflows
• Dynamic Task Composition
• Heterogeneous Platforms
• State-Transfer Awareness