ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

2026-08-25

AI Summary

本文提出ADeptS-Bench基准，通过安全性和消歧两个方面评估计算机使用代理的安全交互能力，揭示了现有模型在处理恶意任务和模糊指令时的不足。

Abstract

Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguous instructions. We introduce ADeptS-Bench, a dual-stream trustworthiness benchmark, grounded in the ADEPTS capability framework and general population user studies. The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface. The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous. Evaluating seven models reveals that no model consistently exceeds 80% task success while staying below 30% attack success; every model clicks "Checkout" on a $25K order without hesitation, and none detects that a "factory reset" button is mislabeled as "Optimize." An ablation reveals three distinct safety architectures: tool-dependent (ASR +21-23pp without refusal tool), partially tool-dependent (+10-11pp), and no mechanism (unchanged). In disambiguation, all models overestimate consequence severity, mirroring the over-refusal bias observed in safety. We release all data, evaluation code, and analysis tools upon publication.

Problem

• Computer Use Agents
• trustworthiness
• visual interfaces
• ambiguous instructions
• benchmark

Innovation

• Dual-stream Trustworthiness Benchmark
• Safety and Disambiguation Streams
• ADEPTS Capability Framework