Computer Use at the Edge of the Statistical Precipice

2026-05-07

AI Summary

This study addresses the pervasive overestimation of Computer-Using Agent (CUA) performance in existing evaluations, which stems from poorly designed environments and statistically unreliable methodologies. To rectify this, the work systematically uncovers the issue for the first time and introduces PRISM—a set of five principled guidelines for robust environment design. It further presents DigiWorld, a sandboxed benchmark comprising 15 mobile applications and over 3.2 million verified configurations. Complementing this, the authors propose a statistically sound evaluation framework based on Wilson confidence intervals and hierarchical bootstrap resampling to enable trustworthy assessment of CUA capabilities. Empirical results demonstrate that conventional static benchmarks are easily outperformed by trivial replay scripts, thereby underscoring the necessity and efficacy of the proposed evaluation paradigm.

Abstract

Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically address. We show that a 1MB replay script that blindly executes a recorded action sequence without ever observing the screen outperforms frontier models on prominent static benchmarks, and prove that its expected success rate is exactly equal to the source agent's pass@k in deterministic environments. We trace this and other failures to two root causes: non-principled environment design (static, unsandboxed, or unreliably verified environments) and non-principled evaluation methodology (naive aggregation and misuse of pass@k for stateful UI interactions). To address the first, we propose PRISM, five design principles for CUA environments (privileged verification, realistic environments, integrity-checked configurations, sandboxed execution, and multifactorial variability) and instantiate them in DigiWorld, a benchmark of 15 realistic sandboxed mobile applications able to evaluate agents in over 3.2 million verified unique configurations. To address the second, we develop an aggregation framework pairing Wilson score intervals with hierarchical bootstrap, producing confidence intervals that correctly account for the nested structure of CUA benchmarks, as we empirically demonstrate. All together, we show that principled environment design and rigorous evaluation methodology are not optional refinements but prerequisites for meaningful CUA research.

Problem

• Computer Use Agents
• evaluation methodology
• environment design
• pass@k
• benchmarking

Innovation

• Computer Use Agents
• PRISM principles
• DigiWorld benchmark
• evaluation methodology
• hierarchical bootstrap