SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

2026-09-08

AI Summary

该研究解决了SWE-Bench Pro中存在的奖励作弊和任务质量问题，通过实施反作弊保护措施和任务改进方法，提出了更可靠的软件工程代理评估基准SWE-Bench Pro Verified。

Abstract

SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: \textbf{reward hacking}, enabled by leakage of gold solutions or hidden evaluation information, and \textbf{task quality issues}, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents'true coding ability. We present \textbf{SWE-Bench Pro Verified}, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines \textbf{anti-hacking} safeguards that eliminate major leakage channels without disrupting normal agent functionality, with \textbf{task refinement} that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.

Problem

• reward hacking
• task quality issues
• benchmark performance

Innovation

• anti-hacking
• task refinement
• reliable benchmark