On the Reliability of Computer Use Agents

2026-04-20

AI Summary

This study addresses the instability of AI agents when repeatedly executing identical tasks, where failures can occur even after prior successes despite unchanged models and environments. For the first time, it systematically identifies and quantifies three key sources undermining reliability: execution stochasticity, ambiguity in task descriptions, and behavioral variability of agents. Leveraging the OSWorld platform, the authors develop a repeated-task execution framework that integrates paired statistical tests with behavioral variation analysis, revealing that both task specification clarity and cross-run behavioral consistency jointly determine agent reliability. The work advocates evaluating agents under repeated execution and proposes enhancing reliability through interactive disambiguation and stability-promoting strategies across runs, offering empirical foundations and design principles for building highly reliable intelligent agents.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Multiagent Systems under Uncertainty
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.

Problem

• reliability
• computer-use agents
• task execution
• stochasticity
• task ambiguity

Innovation

• computer-use agents
• reliability
• task ambiguity
• execution stochasticity
• behavior variability