Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations

2025-01-27

AI Summary

GitHub Actions failure logs are verbose and obscure, significantly impeding fault localization and remediation. Method: This paper presents the first systematic evaluation of large language models (LLMs) for interpreting real-world CI/CD logs, introducing an experience-aware explanation generation paradigm tailored to developers’ expertise levels. We optimize mainstream LLMs (e.g., GPT, Claude) via prompt engineering and conduct empirical user studies, expert evaluations, and qualitative analysis. Contribution/Results: Over 80% of developers deem LLM-generated explanations accurate and actionable for common errors. Novices prefer step-by-step remediation guidance, whereas experienced developers prioritize concise root-cause identification. LLMs demonstrate robust performance on single-step failures but struggle with multi-stage, dependency-heavy pipelines due to limited causal reasoning. Our findings establish a methodological foundation and empirical evidence for integrating LLMs into intelligent DevOps diagnostics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

GitHub Actions (GA) has become the de facto tool that developers use to automate software workflows, seamlessly building, testing, and deploying code. Yet when GA fails, it disrupts development, causing delays and driving up costs. Diagnosing failures becomes especially challenging because error logs are often long, complex and unstructured. Given these difficulties, this study explores the potential of large language models (LLMs) to generate correct, clear, concise, and actionable contextual descriptions (or summaries) for GA failures, focusing on developers' perceptions of their feasibility and usefulness. Our results show that over 80% of developers rated LLM explanations positively in terms of correctness for simpler/small logs. Overall, our findings suggest that LLMs can feasibly assist developers in understanding common GA errors, thus, potentially reducing manual analysis. However, we also found that improved reasoning abilities are needed to support more complex CI/CD scenarios. For instance, less experienced developers tend to be more positive on the described context, while seasoned developers prefer concise summaries. Overall, our work offers key insights for researchers enhancing LLM reasoning, particularly in adapting explanations to user expertise.

Problem

• GitHub Actions
• Error Interpretation
• Development Efficiency

Innovation

• Large Language Models
• GitHub Actions Error Interpretation
• Developer Efficiency Improvement