Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations

2025-07-22

AI Summary

This study investigates the feasibility of leveraging large language models (LLMs) to automatically generate GitHub Actions CI configurations, addressing the labor-intensive and error-prone nature of manual YAML authoring. We introduce the first annotated CI configuration dataset derived systematically from official documentation and conduct a comprehensive zero-shot evaluation of six state-of-the-art models—including GPT-4o, Llama, and Gemma—on the natural-language-to-YAML generation task. Results show that general-purpose LLMs significantly outperform code-specialized models; the highest structural similarity reaches 69%, yet only 3% of generated configurations are fully executable. Primary failure modes include missing workflow steps and semantic misinterpretations of CI semantics. The study uncovers a critical gap between semantic understanding and structural precision in current LLMs for CI configuration generation, establishing a new benchmark and empirical foundation for evaluating and enhancing LLM capabilities in DevOps-oriented automation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Continuous Integration (CI) services, such as GitHub Actions, require developers to write YAML-based configurations, which can be tedious and error-prone. Despite the increasing use of Large Language Models (LLMs) to automate software engineering tasks, their ability to generate CI configurations remains underexplored. This paper presents a preliminary study evaluating six LLMs for generating GitHub Actions configurations from natural language descriptions. We assess three general-purpose foundation models (GPT-4o, Llama, and Gemma) and three code-pretrained models (GPT-4.1, Code Llama, and CodeGemma). We also introduce the first labeled dataset of its kind, constructed from GitHub Actions documentation, pairing descriptions with corresponding best-practice YAML configurations. Zero-shot prompting achieves up to 69% similarity with the ground truth, with only 3% perfect matches. Code-pretrained models slightly underperform compared to general-purpose ones in YAML-based CI tasks, revealing LLM limitations for CI configuration generation. Analyzing GPT-4o outputs reveals issues like missing or renamed steps, misinterpreted descriptions, and unnecessary additions that may affect structural and contextual correctness, indicating a gap between generation quality and the precision required for executable CI configurations. Our research offers insights for improving LLM alignment with configuration languages and guiding future efforts on CI automation and tooling support.

Problem

• Evaluating LLMs for generating GitHub Actions CI configurations
• Assessing accuracy and limitations of LLMs in YAML-based CI tasks
• Identifying gaps in LLM-generated CI configurations' structural correctness

Innovation

• Evaluates six LLMs for GitHub Actions generation
• Introduces first labeled dataset for CI configurations
• Analyzes GPT-4o output issues in CI tasks