A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems

2026-04-22

AI Summary

This work addresses the limitations in evaluating automated vulnerability detection tools, which stem from heterogeneous vulnerability data sources, inconsistent identifiers, and ambiguous version ranges. Leveraging the Open Source Vulnerabilities (OSV) database, we construct a standardized, cross-ecosystem benchmark dataset through precise version mapping and systematic data curation. We propose a reproducible methodology for dataset construction and release an open-source toolkit that enables on-demand reconstruction of historical snapshots, significantly enhancing the transparency and reproducibility of evaluations. Experimental results reveal systematic performance disparities among widely used vulnerability detection tools, underscoring the critical role of high-quality benchmarks in the rigorous assessment of security analysis tools.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Automated vulnerability detection tools are widely used to identify security vulnerabilities in software dependencies. However, the evaluation of such tools remains challenging due to the heterogeneous structure of vulnerability data sources, inconsistent identifier schemes, and ambiguities in version range specifications. In this paper, we present an empirical evaluation of vulnerability detection across multiple software ecosystems using a curated ground-truth dataset derived from the Open Source Vulnerabilities (OSV) database. The dataset explicitly maps vulnerabilities to concrete package versions and enables a systematic comparison of detection results across different tools and services. Since vulnerability databases such as OSV are continuously updated, the dataset used in this study represents a snapshot of the vulnerability landscape at the time of the evaluation. To support reproducibility and future studies, we provide an open-source tool that automatically reconstructs the dataset from the current OSV database using the methodology described in this paper. Our evaluation highlights systematic differences between vulnerability detection systems and demonstrates the importance of transparent dataset construction for reproducible empirical security research.

Problem

• vulnerability detection
• evaluation
• ground truth
• software ecosystems
• reproducibility

Innovation

• ground-truth dataset
• vulnerability detection
• software ecosystems
• reproducible evaluation
• OSV database