Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair

2026-08-13

AI Summary

This study empirically uncovers, for the first time, a phenomenon of security degradation induced by large language models (LLMs) during iterative repair of Infrastructure-as-Code (IaC). Analyzing 5,968 repair scenarios and 4,440 iterations, the authors track security state changes using Checkov, terraform validate, and the CIS Benchmark (30 rules) under both standard and strict detection modes. Distinguishing genuine security regressions from multi-resource measurement artifacts, they find that 79% of regressions stem from resource refactoring. Security regressions occur in 24.8% of iterations under standard mode (5.2% under strict mode), with 36.6% self-resolving within an average of 1.2 iterations. Based on these findings, the study identifies the third iteration as the optimal stopping point and formulates design principles for security-aware feedback mechanisms.

Abstract

Background: Iterative feedback loops are the dominant paradigm for improving LLM-generated Infrastructure-as-Code (IaC): validators such as Checkov and terraform validate feed error signals back for successive repair attempts. Prior work reports cumulative-best metrics, which are non-decreasing by construction, so the raw per-iteration security trajectory has never been examined for IaC. Aims: We study security regression (a previously-passing CIS Benchmark check that fails after a repair iteration) to determine whether and how often iterative LLM repair degrades security while fixing other issues. Method: We analyze 5,968 scenario timelines from the IaC-Eval benchmark, each one scenario run through one configuration for up to 5 repair iterations. The 15 configurations (six model-specific RAG, nine model-aggregated non-RAG, three temperatures each) yield 4,440 iteration transitions with Checkov data on both sides. We track 30 individual CIS check IDs and classify root causes from code diffs, under two detection modes: standard (inclusive) and strict (exclusive check failures only). Results: Under standard detection, 13.8% of scenarios (24.8% of transitions) exhibit at least one regression. Under strict detection the rate falls to 3.3% of scenarios (5.2% of transitions), indicating most apparent regressions are multi-resource measurement artifacts. Resource restructuring (79.0%) is the dominant root cause. Regression transitions show 2.6x more code churn (Cohen's d=0.90) and 4.9x higher strict-mode check volatility (d=1.49). Of standard-mode regressions, 36.6% self-correct within an average of 1.2 iterations; iteration 3 is the optimal stopping point. Conclusions: Iterative IaC repair does introduce security regressions, but the conservative, defensible rate is about 3.3% of scenarios. Our findings motivate security-aware feedback-loop design and actionable iteration-budget guidance.

Problem

• security regression
• Infrastructure-as-Code
• LLM repair
• iterative feedback
• CIS Benchmark

Innovation

• security regression
• iterative repair
• Infrastructure-as-Code
• LLM feedback loop
• CIS Benchmark