A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

2026-06-28

AI Summary

Existing benchmarks for microservice fault diagnosis focus solely on final answers, overlooking the systematic reasoning processes of large language model agents. This work proposes the first evaluation paradigm centered on the diagnostic reasoning process, introducing AIOps2025 and RCA100—large-scale, expert-annotated datasets encompassing three key dimensions: fault localization, identification, and root cause attribution. These datasets integrate multimodal observability data with causal reasoning analysis and cover over 500 real-world failure cases. The benchmark’s effectiveness has been validated through an international competition involving more than 6,000 teams, establishing it as the first reasoning-oriented benchmark for intelligent microservice fault diagnosis to be empirically validated at scale.

Abstract

LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data. However, existing benchmarks remain largely outcome-oriented: they score only the final answer and fail to assess the systematic reasoning process in failure diagnosis. We address this gap by introducing two large-scale datasets (AIOps2025 and RCA100) under a reasoning-process evaluation paradigm that assesses agentic diagnostic capability along three dimensions: Localization (where the fault occurs), Identification (what type of fault it is), and Reason (whether the reasoning trace is grounded in relevant evidence). Together, the two datasets comprise over 500 expert-labeled failure cases across two representative microservice systems (HipsterShop and the OpenTelemetry Demo Store). They cover diverse fault scenarios across resource, network, runtime, middleware/database, and application-logic categories and provide fine-grained causal evidence to support agent learning and reasoning-process evaluation. Beyond scale and coverage, the datasets have been carefully labelled by domain experts and validated through large-scale competitions, supporting more than 6,000 participating teams. This makes them not only expert-labeled diagnostic datasets, but also competition-validated benchmarks for evaluating agentic failure diagnosis in real-world microservice environments. Datasets are available at https://www.aiops.cn/gitlab/aiops-live-benchmark/agenticopseval.

Problem

• LLM agents
• microservice failure diagnosis
• reasoning-process evaluation
• observability data
• benchmark

Innovation

• LLM agents
• failure diagnosis
• reasoning-process evaluation
• microservice observability
• expert-labeled benchmark