Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs

2025-10-13

AI Summary

Existing automated program repair research for C/C++ is hindered by the lack of high-quality, open-source benchmarks. Method: We introduce Defects4C—the first large-scale, executable repair benchmark for C/C++—comprising 248 defective functions and 102 vulnerable functions, all drawn from real-world open-source projects and accompanied by reproducible test cases. Constructed from 9 million defect-related commit records, the dataset underwent rigorous manual curation and automated processing, with reproducibility validated via a standardized testing framework. Contribution/Results: Leveraging Defects4C, we conduct a systematic evaluation of 24 state-of-the-art large language models on C/C++ repair tasks, revealing their capabilities and critical limitations. Defects4C enables model fine-tuning and fair, standardized comparison, thereby filling a fundamental gap in benchmarking for C/C++ program repair and advancing the field.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++. To address this issue, we introduce Defects4C, a comprehensive and executable benchmark specifically designed for C/C++ program repair. Our dataset is constructed from real-world C/C++ repositories and includes a large collection of bug-relevant commits (9M in total), 248 high-quality buggy functions, and 102 vulnerable functions, all paired with test cases for reproduction. These resources enable rigorous evaluation of repair techniques and support the retraining of learning-based approaches for enhanced performance. Using Defects4C, we conduct a comprehensive empirical study evaluating the effectiveness of 24 state-of-the-art large language models (LLMs) in repairing C/C++ faults. Our findings offer valuable insights into the strengths and limitations of current LLM-based APR techniques in this domain, highlighting both the need for more robust methods and the critical role of Defects4C in advancing future research

Problem

• Addressing the lack of C/C++ benchmarks for program repair
• Evaluating large language models' capability to fix C/C++ bugs
• Providing real-world bug data to support APR research

Innovation

• Defects4C provides executable C/C++ benchmark
• Includes real-world buggy and vulnerable functions
• Evaluates 24 LLMs for automated program repair