Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback

2026-07-01

AI Summary

Traditional evaluation of code generation and repair relies solely on binary correctness, overlooking critical dimensions such as incremental improvement, feedback utilization, and behavioral preservation during the repair process. To address this limitation, this work proposes PAIR-Bench, a novel benchmark that guides large language models through feedback-driven iterative repair via a structured feedback protocol combining failure-region control and prompt-depth control. The benchmark introduces progressive prompting, grouping of failure scenarios, and multi-level repair guidance, alongside trajectory-level evaluation metrics. For the first time, it enables fine-grained, adaptive assessment of the code improvement process, outperforming conventional coarse-grained evaluation paradigms in terms of target repair efficacy, generalization capability, behavioral consistency, and required human assistance.

Abstract

Large language models (LLMs) are typically evaluated on code generation and program repair using binary functional correctness: a generated program or patch either passes or fails a test suite. This protocol is simple but coarse, as it ignores partial progress, feedback use, regressions, and the refinement trajectory through which models often improve code. We introduce PAIR-Bench, a progressive and adaptive benchmark for evaluating code improvement: transforming an incorrect or incomplete program into a more correct one through feedback-guided refinement. PAIR-Bench uses progressive hinting, a structured feedback protocol with two controls. Failure-region control determines what the feedback targets by grouping hidden failing tests into failure scenarios, while hint-depth control determines how much repair-relevant information is revealed, from coarse symptoms to implementation-level guidance. This design enables PAIR-Bench to measure whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires. By evaluating repair trajectories progressive metrics rather than only final pass/fail outcomes, PAIR-Bench provides a finer-grained assessment of LLM code-improvement capability.

Problem

• code improvement
• large language models
• program repair
• evaluation benchmark
• feedback-guided refinement

Innovation

• progressive hinting
• adaptive feedback
• code improvement
• failure-region control
• hint-depth control