CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

2025-07-25

AI Summary

To address the limited generalization capability of single models in AIOps and their inability to collaboratively handle heterogeneous operational tasks—ranging from low-level code generation to high-level root-cause analysis—this paper proposes CoE-Ops, a large language model (LLM)-based expert collaboration framework. CoE-Ops integrates fine-grained task classification routing with retrieval-augmented generation (RAG) to enable cross-task knowledge orchestration and answer generation. Its core innovation lies in establishing an extensible, collaborative modeling paradigm—not merely model ensembling. Evaluated on the DevOps-EVAL benchmark, CoE-Ops achieves a 72% improvement in high-level task routing accuracy and an 8% absolute gain in end-to-end problem-solving accuracy over single-model baselines, outperforming state-of-the-art large-scale Mixture-of-Experts (MoE) approaches by 14%.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework's capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy.

Problem

• Addresses limitations of single AIOps models in handling diverse tasks
• Proposes a multi-model collaboration framework for AIOps question-answering
• Enhances accuracy in DevOps task routing and problem resolution

Innovation

• LLM-based expert collaboration framework
• Retrieval-augmented generation for QA tasks
• General-purpose LLM task classifier