From Benchmark Data To Applicable Program Repair: An Experience Report

2025-08-21

AI Summary

This work addresses the industrial ineffectiveness of automated program repair (APR) and the misalignment between academic benchmarks and real-world defects. We propose a novel approach that leverages formal specifications—expressed in the Java Modeling Language (JML)—to guide large language models (LLMs) in generating high-coverage unit tests. By embedding specifications into the test-generation pipeline and integrating contract automata with human feedback, our method significantly improves detection of logic errors, string manipulations, and exception-handling edge cases. Evaluation shows that our technique outperforms state-of-the-art APR methods on standard benchmarks; JML specifications prove particularly valuable for exposing complex semantic errors; yet passing generated tests does not guarantee repair correctness. The study reveals a fundamental gap between academic evaluation criteria and industrial requirements, offering both a scalable technical pathway and empirical evidence toward practical, specification-guided program repair.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper describes our approach to automated program repair. We combine various techniques from the literature to achieve this. Our experiments show that our approach performs better than other techniques on standard benchmarks. However, on closer inspection, none of these techniques work on realistic defects that we see in industry. We find that augmenting code with formal specifications enables LLMs to generate higher-quality unit tests, especially for complex production code with improved coverage of edge cases and exception handling. However, specifications add little value for well-understood errors (e.g., null pointer, index out of bounds), but are beneficial for logic and string manipulation errors. Despite encouraging benchmark results, real-world adoption is limited since passing tests do not guarantee correct patches. Current challenges include insufficient expressiveness of the JML specification language, necessitating advanced verification tools and richer predicates. Our ongoing work is exploring contract automata, programming by example, and testcase repair, with a focus on integrating human feedback and measuring productivity gains - highlighting the gap between academic benchmarks and practical industry needs

Problem

• Addressing the gap between academic benchmarks and practical industry repair needs
• Improving automated program repair with formal specifications for complex code
• Enhancing unit test generation quality and coverage for realistic defects

Innovation

• Augmenting code with formal specifications for LLMs
• Generating higher-quality unit tests for production code
• Exploring contract automata and programming by example