Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

2024-09-25

arXiv.org

AI Summary

To address the high latency and low reliability of multimodal large language model (MLLM)-based agents interacting via graphical user interfaces (GUIs), this paper proposes AXIS, an API-first intelligent agent framework that establishes a novel human–agent–computer collaboration paradigm. Methodologically, AXIS introduces: (1) the Human–Agent–Computer Interaction (HACI) paradigm and an API-first agent architecture; (2) a UI-exploration-driven approach for automatic API discovery and abstraction, enabling “API-ification” of desktop applications; and (3) a principled roadmap for Agent OS evolution and LLM-era GUI design guidelines. Experiments on Microsoft Office Word tasks demonstrate that AXIS reduces execution time by 65–70% and user cognitive load by 38–53% compared to GUI-based baselines, while achieving 97–98% task accuracy.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent Communication

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Office Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compare to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and a fresh UI design principle for application providers in the era of LLMs. It also explores the possibility of turning every applications into agents, paving the way towards an agent-centric operating system (Agent OS).

Problem

• Reduces latency in LLM-based agent interactions with UIs
• Improves reliability of agents by prioritizing API actions
• Automates API creation and expansion for application integration

Innovation

• API-first LLM agents reduce UI interaction latency
• Automated API creation enhances application exploration
• AXIS framework improves efficiency and cognitive workload