AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants

2025-01-27

AI Summary

This paper addresses the challenge of enabling instruction-driven Computer Control Agents (CCAs) to perform fine-grained GUI operations—such as clicking, text input, and instruction understanding—and coordinate multi-task workflows across heterogeneous devices (e.g., PCs and smartphones). We propose the first formal problem definition for CCAs and introduce a unified three-dimensional taxonomy encompassing environment, interaction, and agent design. Through a systematic survey of 86 CCA agents and 33 benchmark datasets, we identify transferable insights from specialized agents—particularly in environment modeling—to large language model (LLM)/vision-language model (VLM)-driven agents. We advocate a paradigm shift from rule-based engines toward integrated LLM/VLM architectures that jointly handle screen/HTML parsing, action modeling, and executable code generation. Our analysis clarifies evaluation criteria and deployment bottlenecks, establishing a theoretical framework and practical guidelines for developing robust, general-purpose, and trustworthy desktop-level AI assistants.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Humans and AI: Interaction Techniques and Devices

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Instruction-based computer control agents (CCAs) execute complex action sequences on personal computers or mobile devices to fulfill tasks using the same graphical user interfaces as a human user would, provided instructions in natural language. This review offers a comprehensive overview of the emerging field of instruction-based computer control, examining available agents -- their taxonomy, development, and respective resources -- and emphasizing the shift from manually designed, specialized agents to leveraging foundation models such as large language models (LLMs) and vision-language models (VLMs). We formalize the problem and establish a taxonomy of the field to analyze agents from three perspectives: (a) the environment perspective, analyzing computer environments; (b) the interaction perspective, describing observations spaces (e.g., screenshots, HTML) and action spaces (e.g., mouse and keyboard actions, executable code); and (c) the agent perspective, focusing on the core principle of how an agent acts and learns to act. Our framework encompasses both specialized and foundation agents, facilitating their comparative analysis and revealing how prior solutions in specialized agents, such as an environment learning step, can guide the development of more capable foundation agents. Additionally, we review current CCA datasets and CCA evaluation methods and outline the challenges to deploying such agents in a productive setting. In total, we review and classify 86 CCAs and 33 related datasets. By highlighting trends, limitations, and future research directions, this work presents a comprehensive foundation to obtain a broad understanding of the field and push its future development.

Problem

• AI Assistants
• Complex Environments
• Multi-task Evolution

Innovation

• Computer Control Assistants
• Multifunctional CCAs
• Large Language Models