Enhancing Reliability in LLM-Based Secure Code Generation

2026-05-22

AI Summary

This work addresses the insufficient security and reliability of large language models (LLMs) in code generation, particularly their inconsistent ability to avoid vulnerabilities across diverse programming languages and prompting strategies. To tackle this challenge, the authors propose the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which uniquely integrates Common Weakness Enumeration (CWE) mitigation knowledge into chain-of-thought reasoning. MA-CoT further incorporates a language-aware security guardrail and a hierarchical attribution method for vulnerability root causes. Evaluated on GPT-5, Claude-4.5, and Gemini-2.5 across C, Java, and Python, MA-CoT reduces security issues by 57.6% and 94.5% on the main dataset and LLMSecEval benchmark, respectively, with high-severity vulnerabilities decreasing by 56.7% and 95.6%, substantially outperforming existing prompting approaches.

Abstract

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the \textit{Mitigation-Aware Chain-of-Thought (MA-CoT)} framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6\%) on the primary dataset and from 73 to 4 (94.5\%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7\%) and from 45 to 2 (95.6\%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Problem

• LLM-based code generation
• security reliability
• prompting strategies
• code vulnerabilities
• CWE mitigation

Innovation

• Mitigation-Aware Chain-of-Thought
• secure code generation
• CWE mitigation
• LLM security reliability
• vulnerability attribution