SoK: Automated Vulnerability Repair: Methods, Tools, and Assessments

2025-06-13

AI Summary

The rapid proliferation of software vulnerabilities necessitates efficient Automated Vulnerability Repair (AVR) techniques, yet existing approaches suffer from the absence of a unified evaluation benchmark and systematic analysis. To address this, we introduce Vul4C—the first large-scale, real-world benchmark for C/C++ vulnerability repair—comprising 144 vulnerabilities, each accompanied by exploit code and manually validated patches. We further propose a comprehensive, three-stage AVR evaluation framework covering vulnerability analysis, patch generation, and patch verification. Leveraging static/dynamic analysis, program synthesis, symbolic execution, and test-driven validation, we conduct standardized evaluations of seven C/C++ and two Java AVR tools. Experimental results expose critical bottlenecks in current tools, including low real-world repair rates and poor generalization capability. Vul4C establishes a reproducible, comparable benchmark platform and provides empirical foundations to advance AVR research.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Computer Vision: Adversarial Attacks & Robustness
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The increasing complexity of software has led to the steady growth of vulnerabilities. Vulnerability repair investigates how to fix software vulnerabilities. Manual vulnerability repair is labor-intensive and time-consuming because it relies on human experts, highlighting the importance of Automated Vulnerability Repair (AVR). In this SoK, we present the systematization of AVR methods through the three steps of AVR workflow: vulnerability analysis, patch generation, and patch validation. We assess AVR tools for C/C++ and Java programs as they have been widely studied by the community. Since existing AVR tools for C/C++ programs are evaluated with different datasets, which often consist of a few vulnerabilities, we construct the first C/C++ vulnerability repair benchmark dataset, dubbed Vul4C, which contains 144 vulnerabilities as well as their exploits and patches. We use Vul4C to evaluate seven AVR tools for C/C++ programs and use the third-party Vul4J dataset to evaluate two AVR tools for Java programs. We also discuss future research directions.

Problem

• Automated repair of software vulnerabilities efficiently
• Systematizing AVR methods: analysis, generation, validation
• Evaluating AVR tools using benchmark datasets

Innovation

• Systematizes AVR workflow steps: analysis, generation, validation
• Constructs Vul4C benchmark with 144 C/C++ vulnerabilities
• Evaluates seven C/C++ and two Java AVR tools