A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

2026-08-09

AI Summary

Current evaluations of code-generating agents focus solely on whether the final patch passes tests, offering limited insight into where and why failures occur during intermediate reasoning steps. This work proposes SWE-RPG, a novel benchmark that introduces human-authored reference solutions encompassing two critical phases—requirement clarification and implementation planning—alongside executable patch evaluation. This enables fine-grained, traceable, and alignment-aware diagnosis of agent reasoning chains. A systematic evaluation across 163 Python and Java repository tasks involving prominent agents such as Claude Code, Codex, and OpenCode reveals an average pass rate of only 31.5%. Diagnostic analysis further identifies implicit requirement recovery as the primary bottleneck, accounting for 24.5%–46.0% of failures.

Abstract

Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.

Problem

• requirement clarification
• implementation planning
• coding agents
• repository-level benchmark
• implicit requirements

Innovation

• requirement clarification
• implementation planning
• coding agents
• repository-level benchmark
• ground-truth diagnosis