Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation

2026-09-15

AI Summary

论文提出一种框架明确指定基于大语言模型的程序修复实验设置，通过分析Defects4J和SWE-bench上的系统，解决了实验设置不透明导致的结果可比性问题。

Abstract

Evaluations of automated program repair (APR) systems commonly report the benchmark, the number of repaired defects, and the tests used for final patch validation, but these items no longer fully specify the repair task presented to a system. Recent LLM-based systems differ in the information supplied before repair, the repository and testing operations permitted during repair, and the feedback returned after unsuccessful attempts, allowing the same benchmark to instantiate substantially different repair tasks ranging from localized patch generation to repository-level diagnosis and iterative repair. We present a framework for explicitly specifying the experimental settings associated with reported APR results. We analyze reported experimental settings from systems evaluated on Defects4J and SWE-bench and characterize each result by its task unit, fault-localization assumptions, initial input, tool access, repair-time feedback, final validation, and resource budget. Our analysis shows that benchmark identity alone is insufficient to reconstruct the evaluated task or determine the appropriate scope of comparison across reported repair rates. We therefore introduce a machine-readable schema for specifying each experimental setting to improve reproducibility and make the scope of cross-system comparisons explicit.

Problem

• Automated Program Repair
• Experimental Settings
• LLM-based Systems
• Reproducibility
• Cross-system Comparisons

Innovation

• Experimental Settings
• Automated Program Repair (APR)
• Machine-Readable Schema
• Reproducibility
• Cross-System Comparisons