A Survey on AgentOps: Categorization, Challenges, and Future Directions

2025-08-04

AI Summary

To address the frequent runtime anomalies in large language model (LLM)-based agent systems and the absence of systematic operational methodologies, this paper presents the first comprehensive survey on AgentOps—the operations of intelligent agent systems. Through systematic literature review and anomaly taxonomy modeling, we formally define internal anomalies (e.g., reasoning hallucinations, tool invocation failures) and external anomalies (e.g., API outages, environmental changes). Building upon this taxonomy, we propose a full-lifecycle AgentOps framework encompassing monitoring, anomaly detection, root-cause analysis, and autonomous recovery. This work establishes the first structured conceptual foundation for agent operations, fills a critical theoretical gap in the field, identifies key open challenges, and provides an extensible methodological basis and evolutionary roadmap for both academic research and industrial deployment.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

As the reasoning capabilities of Large Language Models (LLMs) continue to advance, LLM-based agent systems offer advantages in flexibility and interpretability over traditional systems, garnering increasing attention. However, despite the widespread research interest and industrial application of agent systems, these systems, like their traditional counterparts, frequently encounter anomalies. These anomalies lead to instability and insecurity, hindering their further development. Therefore, a comprehensive and systematic approach to the operation and maintenance of agent systems is urgently needed. Unfortunately, current research on the operations of agent systems is sparse. To address this gap, we have undertaken a survey on agent system operations with the aim of establishing a clear framework for the field, defining the challenges, and facilitating further development. Specifically, this paper begins by systematically defining anomalies within agent systems, categorizing them into intra-agent anomalies and inter-agent anomalies. Next, we introduce a novel and comprehensive operational framework for agent systems, dubbed Agent System Operations (AgentOps). We provide detailed definitions and explanations of its four key stages: monitoring, anomaly detection, root cause analysis, and resolution.

Problem

• Addressing anomalies in LLM-based agent systems for stability
• Establishing a framework for agent system operations (AgentOps)
• Categorizing and resolving intra-agent and inter-agent anomalies

Innovation

• Defines intra-agent and inter-agent anomalies
• Introduces AgentOps operational framework
• Details four key stages of AgentOps