VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

2026-08-12

AI Summary

Existing vulnerability detection benchmarks suffer from limited language coverage, insufficient patch complexity, and inadequate project diversity, hindering comprehensive evaluation of tool performance. To address this, this work introduces VICBench, a multilingual benchmark of vulnerability-introducing commits (VICs) spanning 88 projects in Python, Java, and C++. It comprises 100 CVE-associated VICs validated through both expert review and an agent-based workflow, covering 48 distinct CWE types, with an average fix size of 38.6 lines and corresponding VICs averaging 252.5 lines. As the first large-scale, high-complexity, multilingual dataset of real-world VICs, VICBench substantially enhances evaluation robustness. Benchmarking state-of-the-art methods V-SZZ and LLM4SZZ on VICBench yields F1 scores of only 33.3%–40.1%, underscoring the continued heavy reliance of current approaches on manual intervention.

Abstract

Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.

Problem

• vulnerability detection
• benchmark dataset
• vulnerability-inducing commits
• multi-language coverage
• CVE

Innovation

• vulnerability-inducing commits
• multi-language benchmark
• dual annotation
• real-world vulnerability fixes
• code vulnerability detection