What Makes a Good Bug Report for an AI Agent?

2026-07-08

AI Summary

This study addresses the unresolved question of whether existing human-oriented bug report standards are suitable for large language model (LLM)-driven automated program repair. It presents the first systematic investigation into how 27 report characteristics influence LLM repair performance, employing statistical modeling, controlled ablation studies, and multi-agent comparative experiments to evaluate Qwen and Gemma on the SWE-bench Verified and Pro benchmarks. The findings reveal that executable reproduction scripts, precise fault localization, source code context, and explicit repair suggestions significantly enhance repair success rates, whereas verbose or loosely structured descriptions degrade performance. Moreover, different models exhibit distinct robustness strategies under information scarcity. These results challenge conventional bug reporting guidelines and underscore that high-quality reports for AI agents should prioritize executability, precise localization, and clear structure.

Abstract

Automated program repair (APR) agents are transitioning from research benchmarks to developer workflows, yet they still begin with bug reports written for human developers. While decades of research have established what makes a good bug report for humans (e.g., steps to reproduce, stack traces), it remains unclear whether these features transfer to LLM-based agents. We study this question in two analyses. First, we use statistical modeling to examine associations between 27 bug-report features and repair success across 433 SWE-bench Verified issues attempted by 87 repair agents. We find that fix suggestions, reproduction scripts, repository source code, and localization info are associated with higher resolution likelihood, while longer reports are associated with lower odds. Second, we conduct controlled ablations across 2 models and 17 problem-statement mutations on SWE-bench Pro, varying the information available to an agent while holding the underlying task fixed. We remove or isolate selected bug-report content, delete fault-localization cues, and test structural changes that flatten lists or remove section headers. We find that both models depend on localization cues and expected behavior, and that structural changes alone can reduce solve rates, even without removing any content. The two models diverge in how they handle missing information: Qwen searches more widely and can exhaust its turn budget, while Gemma commits to a plausible interpretation early and patches on it. Our findings indicate that a good bug report for an agent overlaps with, but is not identical to, a good report for a human: agents benefit most from concrete, executable, and well-localized information, whereas some qualities long emphasized for human readers, such as natural language steps to reproduce and readable descriptions, contribute little or even correlate with lower success.

Problem

• bug report
• automated program repair
• LLM-based agent
• repair success
• defect localization

Innovation

• Automated Program Repair
• LLM-based Agents
• Bug Report Optimization
• Fault Localization
• Controlled Ablation Study