Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study

2024-12-24

AI Summary

This work systematically evaluates the effectiveness and limitations of large language models (LLMs) for code vulnerability detection (CVD). Addressing two practical challenges—class imbalance and variable code length—we conduct comprehensive benchmarking across five datasets containing both short and long code samples, evaluating LLaMA, CodeLlama, StarCoder, and Phi series models against five graph neural networks (e.g., GGNN, Code2Vec) and sequence-based models (e.g., BiLSTM). To our knowledge, this is the first empirical study to rigorously assess LLMs on CVD, uncovering their inherent limitations in modeling long code contexts and minority-class vulnerabilities. Experimental results show that fine-tuned LLMs achieve an average 12.3% improvement in F1-score over traditional approaches. To foster reproducibility and advance software security research, we publicly release all code, fine-tuning datasets, and training configurations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Code vulnerability detection (CVD) is essential for addressing and preventing system security issues, playing a crucial role in ensuring software security. Previous learning-based vulnerability detection methods rely on either fine-tuning medium-size sequence models or training smaller neural networks from scratch. Recent advancements in large pre-trained language models (LLMs) have showcased remarkable capabilities in various code intelligence tasks including code understanding and generation. However, the effectiveness of LLMs in detecting code vulnerabilities is largely under-explored. This work aims to investigate the gap by fine-tuning LLMs for the CVD task, involving four widely-used open-source LLMs. We also implement other five previous graph-based or medium-size sequence models for comparison. Experiments are conducted on five commonly-used CVD datasets, including both the part of short samples and long samples. In addition, we conduct quantitative experiments to investigate the class imbalance issue and the model's performance on samples of different lengths, which are rarely studied in previous works. To better facilitate communities, we open-source all codes and resources of this study in https://github.com/SakiRinn/LLM4CVD and https://huggingface.co/datasets/xuefen/VulResource.

Problem

• Large-scale Pre-trained Language Models
• Code Vulnerability Detection
• Imbalanced Data

Innovation

• Large-scale Pre-trained Language Models
• Code Vulnerability Detection
• Imbalanced Datasets and Code Length Impact