A Two-Staged LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation

2025-06-04

AI Summary

CI/CD pipeline failure diagnosis and repair have long suffered from high complexity and low automation. This paper introduces LogSage—the first end-to-end, LLM-driven framework for root-cause analysis (RCA) and automated repair. It features a novel two-stage LLM architecture: (1) Stage I employs intelligent log preprocessing to precisely localize failures; (2) Stage II integrates retrieval-augmented generation (RAG) with tool calling to generate executable, validated fixes. LogSage is the first industrial-grade solution validated on over one million production CI/CD pipelines. It achieves 98% RCA accuracy—12 percentage points higher than state-of-the-art baselines—and end-to-end repair accuracy exceeding 88%. Deployed at scale, it supported 1.07 million CI/CD executions in its first year, processing over 3,000 tasks daily.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Continuous Integration and Continuous Deployment (CI/CD) pipelines are pivotal to modern software engineering, yet diagnosing and resolving their failures remains a complex and labor-intensive challenge. In this paper, we present LogSage, the first end-to-end LLM-powered framework that performs root cause analysis and solution generation from failed CI/CD pipeline logs. During the root cause analysis stage, LogSage employs a specialized log preprocessing pipeline tailored for LLMs, which extracts critical error logs and eliminates noise to enhance the precision of LLM-driven root cause analysis. In the solution generation stage, LogSage leverages RAG to integrate historical resolution strategies and utilizes tool-calling to deliver actionable, automated fixes. We evaluated the root cause analysis stage using a newly curated open-source dataset, achieving 98% in precision and 12% improvement over naively designed LLM-based log analysis baselines, while attaining near-perfect recall. The end-to-end system was rigorously validated in a large-scale industrial CI/CD environment of production quality, processing more than 3,000 executions daily and accumulating more than 1.07 million executions in its first year of deployment, with end-to-end precision exceeding 88%. These two forms of evaluation confirm that LogSage providing a scalable and practical solution to manage CI/CD pipeline failures in real-world DevOps workflows.

Problem

• Detect and fix CI/CD pipeline failures automatically
• Improve precision in root cause analysis of logs
• Integrate historical solutions for actionable fixes

Innovation

• LLM-powered end-to-end CI/CD failure framework
• Specialized log preprocessing for LLM analysis
• RAG and tool-calling for automated fixes