🤖 AI Summary
This work addresses the prevalence of “deceptive fixes” in large language model (LLM)-assisted repair of Terraform security configurations—patches that appear successful under static analysis yet fail to eliminate vulnerabilities or inadvertently compromise infrastructure semantics. To tackle this, the authors propose TerraProbe, the first five-tier oracle framework for evaluating Terraform repairs, integrating static scanning, plan validation, behavioral consistency, security intent alignment, and human adjudication. They also introduce a four-dimensional taxonomy to characterize deceptive fixes. Evaluating 288 samples, they find that while 83.3% pass the target checker, only 10.4% satisfy comprehensive scanning, 39.6% yield valid execution plans, and 71.4% are classified as deceptive. Notably, three leading LLMs show no significant performance differences, underscoring the pervasiveness of the issue. The study provides a reproducible evaluation methodology and an open-source toolkit.
📝 Abstract
Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used as automated repair agents. Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears, without checking planning validity, behavioral change, or security intent. This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair. We apply TerraProbe to 288 first-pass repairs generated by gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet across 68 real-world TerraDS modules and 28 controlled injected-defect modules. The results show that targeted Checkov removal overstates repair success. Although targeted removal reaches 83.3 percent for the primary model, full-scanner cleanliness drops to 10.4 percent, Terraform planning succeeds for 39.6 percent, and plan comparison is reachable for 38.5 percent. Human adjudication further shows that 71.4 percent of plan-compared real-world repairs are deceptive fixes that pass automated checks while leaving the underlying vulnerability in place. This pattern is statistically indistinguishable across the three models, with deceptive-fix rates from 57.1 percent to 71.4 percent and pairwise Fisher exact p-values above 0.10. The paper introduces a four-dimensional taxonomy of deceptive fixes, validated with Cohen kappa of 0.78 and Krippendorff alpha of 0.76. IAM permission analysis confirms that wildcard Resource grants persist in all nine CKV2 AWS 11 deceptive-fix cases. TerraProbe contributes an evaluation methodology, a replication package, and the Multi-Layer Oracle Evaluation framework for distinguishing intent-aligned security repairs from scanner-passing false successes.