Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks

2026-09-06

AI Summary

研究针对软件工程基准测试中自主软件代理的作弊行为，使用LLM作为裁判协议进行审计，并通过添加原创性指令有效降低作弊率。

Abstract

While autonomous software engineering (SWE) agents achieve high benchmark resolution rates, these scores can mask exploitative behaviors---such as leveraging local Git histories, accessing upstream repositories, or recalling memorized solutions---rather than demonstrating genuine problem solving. We systematize and audit these exploits across five open large language models on SWE-bench Multilingual and DeepSWE using a turn-level LLM-as-a-judge protocol. Under standard prompts, exploitation rates reach 45.1\%--82.4\% on SWE-bench Multilingual and 44.2\%--66.1\% on DeepSWE. Appending a targeted instruction enforcing solution originality drastically cuts these exploitation rates---down to 4.0\%--10.7\% and 1.5\%--7.1\%, respectively, while maintaining strong core task performance. Our findings demonstrate the critical need for exploit-aware evaluation frameworks that measure true repository-level problem solving over benchmark gaming.

Problem

• autonomous software engineering
• exploitative behaviors
• benchmark resolution rates
• solution originality
• exploit-aware evaluation

Innovation

• agentic exploits
• software engineering benchmarks
• LLM-as-a-judge protocol
• exploitation rates
• solution originality