Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?

2025-07-29

AI Summary

Current vulnerability detection models exhibit poor out-of-distribution generalization, primarily due to low label accuracy (20–71%), severe data duplication, and insufficient coverage of critical CWEs in existing datasets—leading models to learn spurious correlations. To address this, we propose a systematic solution: (1) BenchVul, a high-confidence benchmark comprising manually annotated samples covering MITRE’s Top 25 CWEs; (2) TitanVul, a large-scale, deduplicated training dataset generated via multi-agent LLM collaboration for precise code understanding and security validation; and (3) RVG, a realistic vulnerability generation framework that simulates development workflows to produce context-aware, scarce vulnerability samples. Experiments show that jointly training on TitanVul and RVG achieves 0.874 accuracy on BenchVul—outperforming baselines by 14.0%—significantly mitigating self-testing bias and generalization failure.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Automated vulnerability detection research has made substantial progress, yet its real-world impact remains limited. Current vulnerability datasets suffer from issues including label inaccuracy rates of 20-71%, extensive duplication, and poor coverage of critical CWE types. These issues create a significant "generalization gap" where models achieve misleading self-testing performance (measured on held-out data from same dataset for training) by exploiting spurious correlations rather than learning true vulnerability patterns. Our analysis reveals that many models experience substantial performance drops of up to 40.6% when evaluated on independent data, sometimes underperforming random guessing. To address these limitations, we present a three-part solution. First, we introduce a manually curated test dataset, BenchVul, covering the MITRE Top 25 Most Dangerous CWEs. Second, we construct a high-quality training dataset, TitanVul, comprising 35,045 functions by aggregating seven public sources and applying deduplication and validation using a novel multi-agent LLM framework. Third, we propose a Realistic Vulnerability Generation (RVG) framework, which synthesizes context-aware vulnerability examples for underrepresented but critical CWE types through simulated development workflows. Our evaluation shows the strengths of each component in closing the generalization gap. First, BenchVul shows the limitations of self-testing: models trained on existing datasets, such as BigVul and PrimeVul, experience performance drops on BenchVul (from 0.776 to 0.519 and from 0.567 to 0.337). Second, training models on TitanVul demonstrates improved generalization, with model performance increasing from 0.584 when evaluated on the same dataset to 0.767 when tested on BenchVul. Third, supplementing TitanVul with RVG-generated data yields further gains, increasing model performance by 14.0% to 0.874.

Problem

• Detect Top 25 CWE weaknesses accurately with LLMs
• Address label inaccuracy and duplication in vulnerability datasets
• Close generalization gap in automated vulnerability detection

Innovation

• Manually curated test dataset BenchVul
• High-quality training dataset TitanVul
• Realistic Vulnerability Generation framework RVG