Real-World Usability of Vulnerability Proof-of-Concepts: A Comprehensive Study

2025-10-21

AI Summary

This study systematically identifies a critical challenge in vulnerability disclosure: low PoC availability in the wild—78.9% of CVEs lack publicly available PoCs, existing reports on average omit 30% of essential reproduction information, and reproducibility failure rates remain high. To address this, we construct the largest empirical PoC dataset to date, comprising 470,000 PoCs. We propose a fine-grained component extraction method and adapt a fine-tuned BERT-NER model to quantitatively assess report completeness. Further, we design a comprehensive evaluation framework integrating multi-platform automated collection with manual reproduction experiments. Our analysis identifies key usability barriers—including ambiguous environment dependencies and missing trigger conditions—and derives actionable, implementable strategies to enhance PoC quality. The work establishes a foundational dataset and methodological framework to support collaborative vulnerability response and automated exploit validation.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The Proof-of-Concept (PoC) for a vulnerability is crucial in validating its existence, mitigating false positives, and illustrating the severity of the security threat it poses. However, research on PoCs significantly lags behind studies focusing on vulnerability data. This discrepancy can be directly attributed to several challenges, including the dispersion of real-world PoCs across multiple platforms, the diversity in writing styles, and the difficulty associated with PoC reproduction. To fill this gap, we conduct the first large-scale study on PoCs in the wild, assessing their report availability, completeness, reproducibility. Specifically, 1) to investigate PoC reports availability for CVE vulnerability, we collected an extensive dataset of 470,921 PoCs and their reports from 13 platforms, representing the broadest collection of publicly available PoCs to date. 2) To assess the completeness of PoC report at a fine-grained level, we proposed a component extraction method, which combines pattern-matching techniques with a fine-tuned BERT-NER model to extract 9 key components from PoC reports. 3) To evaluate the effectiveness of PoCs, we recruited 8 participants to manually reproduce 150 sampled vulnerabilities with 32 vulnerability types based on PoC reports, enabling an in-depth analysis of PoC reproducibility and the factors influencing it. Our findings reveal that 78.9% of CVE vulnerabilities lack available PoCs, and existing PoC reports typically miss about 30% of the essential components required for effective vulnerability understanding and reproduction, with various reasons identified for the failure to reproduce vulnerabilities using available PoC reports. Finally, we proposed actionable strategies for stakeholders to enhance the overall usability of vulnerability PoCs in strengthening software security.

Problem

• Assessing availability and completeness of vulnerability Proof-of-Concept reports
• Evaluating reproducibility challenges in real-world vulnerability PoCs
• Identifying gaps in essential components for effective vulnerability reproduction

Innovation

• Collected 470,921 PoCs from 13 platforms
• Extracted key components using BERT-NER model
• Evaluated reproducibility through manual vulnerability reproduction