Can Developers rely on LLMs for Secure IaC Development?

2026-02-03

AI Summary

This study addresses the limited reliability of current large language models (LLMs) in secure Infrastructure-as-Code (IaC) development, particularly their inability to effectively detect security smells and generate secure code. It presents the first systematic evaluation of GPT-4o and Gemini 2.0 Flash across real-world and synthetic IaC scenarios, employing both general and guided prompt engineering strategies, security smell analysis, and manual assessment. Results demonstrate that guided prompting substantially improves detection performance, identifying 78% and 67% of security smells in Stack Overflow and GitHub datasets, respectively. However, only 7%–17% of the generated code adheres to established security standards, underscoring the current inadequacy of LLMs for dependable secure IaC development. This work establishes an empirical benchmark and offers actionable guidance for prompt design in IaC security automation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We investigated the capabilities of GPT-4o and Gemini 2.0 Flash for secure Infrastructure as Code (IaC) development. For security smell detection, on the Stack Overflow dataset, which primarily contains small, simplified code snippets, the models detected at least 71% of security smells when prompted to analyze code from a security perspective (general prompt). With a guided prompt (adding clear, step-by-step instructions), this increased to 78%.In GitHub repositories, which contain complete, real-world project scripts, a general prompt was less effective, leaving more than half of the smells undetected. However, with the guided prompt, the models uncovered at least 67% of the smells. For secure code generation, we prompted LLMs with 89 vulnerable synthetic scenarios and observed that only 7% of the generated scripts were secure. Adding an explicit instruction to generate secure code increased GPT secure output rate to 17%, while Gemini changed little (8%). These results highlight the need for further research to improve LLMs'capabilities in assisting developers with secure IaC development.

Problem

• LLM
• Infrastructure as Code
• Security Smell
• Secure Code Generation
• IaC Security

Innovation

• LLM
• Infrastructure as Code
• security smells
• guided prompting
• secure code generation