Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

2026-07-30

AI Summary

This work addresses a critical limitation in large language models (LLMs) for program repair: their tendency to treat passing tests as valid verification evidence without assessing whether those tests actually detect the original bug. To tackle this, we propose the BSG-VA framework, which systematically quantifies the validity of verification evidence by replaying validation commands across three code states—original buggy, candidate-patched, and developer-provided correct versions—and classifying each verification event based on its discriminative capability against the defect. Combining test patch extraction, code state replay, and controlled experiments on 110 repair tasks, we find that 46% of positively reported verifications lack discriminative power. Integrating a bug-contrast feedback mechanism reduces the rate of invalid evidence loops by 7.8 percentage points and increases discriminative evidence by 7.4 percentage points, all while maintaining consistent repair success rates.

Abstract

When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.

Problem

• program repair agents
• validation evidence
• test adequacy
• large language models
• software quality

Innovation

• validation evidence
• test adequacy
• program repair agents
• BSG-VA
• controlled experiment