A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection

2025-12-18

AI Summary

Large language models (LLMs) suffer from diminished reliability and robustness in code vulnerability detection due to code obfuscation. Method: We introduce the first unified evaluation framework covering 19 obfuscation techniques across layout, data-flow, and control-flow dimensions, systematically benchmarking 15 LLMs and 2 programming agents on Solidity, C, C++, and Python. Our approach integrates a structured taxonomy of obfuscation, LLM-driven obfuscation implementation, multi-dimensional code attribute modeling, standardized vulnerability benchmarks, and a multi-model consensus evaluation protocol. Contribution/Results: This enables the first empirical, cross-model, cross-language, and cross-vulnerability-type analysis. We uncover a dual effect of obfuscation—simultaneously improving and degrading detection performance—dependent on vulnerability type, code structure, and model architecture. Several critical obfuscation patterns causing model failure are identified, and verifiable pathways for enhancing robustness are proposed.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) are increasingly adopted for code vulnerability detection, their reliability and robustness across diverse vulnerability types have become a pressing concern. In traditional adversarial settings, code obfuscation has long been used as a general strategy to bypass auditing tools, preserving exploitability without tampering with the tools themselves. Numerous efforts have explored obfuscation methods and tools, yet their capabilities differ in terms of supported techniques, granularity, and programming languages, making it difficult to systematically assess their impact on LLM-based vulnerability detection. To address this gap, we provide a structured systematization of obfuscation techniques and evaluate them under a unified framework. Specifically, we categorize existing obfuscation methods into three major classes (layout, data flow, and control flow) covering 11 subcategories and 19 concrete techniques. We implement these techniques across four programming languages (Solidity, C, C++, and Python) using a consistent LLM-driven approach, and evaluate their effects on 15 LLMs spanning four model families (DeepSeek, OpenAI, Qwen, and LLaMA), as well as on two coding agents (GitHub Copilot and Codex). Our findings reveal both positive and negative impacts of code obfuscation on LLM-based vulnerability detection, highlighting conditions under which obfuscation leads to performance improvements or degradations. We further analyze these outcomes with respect to vulnerability characteristics, code properties, and model attributes. Finally, we outline several open problems and propose future directions to enhance the robustness of LLMs for real-world vulnerability detection.

Problem

• Systematically assesses code obfuscation impact on LLM-based vulnerability detection
• Evaluates obfuscation techniques across multiple programming languages and LLMs
• Analyzes conditions where obfuscation improves or degrades detection performance

Innovation

• Systematically categorizes obfuscation techniques into layout, data, control flow
• Implements techniques across four languages using LLM-driven approach
• Evaluates impact on multiple LLMs and coding agents under unified framework