🤖 AI Summary
Existing CUA (Computer-User-Agent) benchmarks overemphasize functional correctness and fail to assess agent reliability in enterprise production environments. Method: We propose UI-CUBE—a first-of-its-kind, enterprise-ready, systematic diagnostic benchmark comprising 226 graded tasks. It integrates UI perturbation, multi-resolution UI testing, and application-state verification to rigorously evaluate architectural deficiencies in memory management, hierarchical planning, and state coordination. Contribution/Results: Experiments reveal that state-of-the-art agents achieve only 67–85% success on simple tasks but plummet to 9–19% on complex enterprise workflows—far below human novices (61.2%), exposing fundamental architectural bottlenecks. UI-CUBE establishes a reproducible, attributable reliability evaluation paradigm for industrial deployment of CUAs.
📝 Abstract
While current Computer Use Agent (CUA) benchmarks measure task completion effectively, they provide limited assessment of enterprise deployment readiness, emphasizing functional correctness over the operational reliability required for production systems. We present UI-CUBE (UiPath Computer Use BEnchmark), a systematic benchmark comprising 226 tasks across two difficulty tiers designed to expose fundamental architectural limitations in current CUAs. Our evaluation covers simple UI interactions (136 tasks) and complex workflows including copy-paste tasks (50 tasks) and enterprise application scenarios (40 tasks), with systematic interface variation coverage, multi-resolution testing and automated validation of task success through the application state. Evaluation of five state-of-the-art models reveals a sharp capability cliff rather than gradual performance degradation. Simple UI interactions achieve 67-85% success rates (compared to 97.9% human performance), but complex workflows drop precipitously to 9-19%. Human evaluators with no prior application experience achieve only 61.2% on complex tasks despite near-perfect performance on simple tasks, establishing realistic performance ceilings. This discontinuous performance pattern -- where agents achieve 68-87% of human performance on simple tasks but only 15-32% on complex workflows -- indicates fundamental architectural limitations in memory management, hierarchical planning, and state coordination rather than incremental capability gaps addressable through better training or prompting. UI-CUBE functions as an enterprise-readiness diagnostic, revealing that while current CUAs can manipulate individual interface elements, they cannot yet function as reliable workflow automation tools. These findings provide architectural insights essential for developing production-ready CUAs capable of managing complex, multi-step enterprise processes.