🤖 AI Summary
Large language models (LLMs) exhibit low correctness and intent alignment when generating Terraform infrastructure-as-code (IaC), hindering reliable automation. Method: We propose a structured knowledge-enhanced approach: (1) establishing the first LLM error taxonomy specifically for IaC, revealing the “correctness–consistency gap”; (2) designing Semantic-Enhanced Graph RAG (SE-Graph RAG) that explicitly models resource dependencies and integrates graph neural networks for graph-structured retrieval-augmented generation; and (3) validating via a cloud-based simulation environment with automated, closed-loop error analysis. Results: Our method increases generation success rate from 27.1% to 75.3%, achieving an overall task success rate of 62.6%. This work provides the first systematic characterization of IaC generation failure modes and empirically demonstrates that architectural-level semantic understanding—not merely syntactic or lexical fidelity—constitutes the fundamental bottleneck in LLM-assisted IaC.
📝 Abstract
Large Language Models (LLMs) currently exhibit low success rates in generating correct and intent-aligned Infrastructure as Code (IaC). This research investigated methods to improve LLM-based IaC generation, specifically for Terraform, by systematically injecting structured configuration knowledge. To facilitate this, an existing IaC-Eval benchmark was significantly enhanced with cloud emulation and automated error analysis. Additionally, a novel error taxonomy for LLM-assisted IaC code generation was developed. A series of knowledge injection techniques was implemented and evaluated, progressing from Naive Retrieval-Augmented Generation (RAG) to more sophisticated Graph RAG approaches. These included semantic enrichment of graph components and modeling inter-resource dependencies. Experimental results demonstrated that while baseline LLM performance was poor (27.1% overall success), injecting structured configuration knowledge increased technical validation success to 75.3% and overall success to 62.6%. Despite these gains in technical correctness, intent alignment plateaued, revealing a "Correctness-Congruence Gap" where LLMs can become proficient "coders" but remain limited "architects" in fulfilling nuanced user intent.