Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

2026-07-01

AI Summary

Current benchmarks for code optimization agents—such as GSO, SWE-Perf, and SWE-efficiency—suffer from evaluation results that are highly sensitive to runtime instability, scoring rule biases, and task aggregation methodologies, thereby failing to accurately reflect agent performance. This work presents the first systematic audit of the reliability of these three major benchmarks by replaying official reference patches across platforms, analyzing the impact of scoring rules on rankings, and retrospectively evaluating publicly submitted solutions. The study reveals that a majority of reference patches fail to reproduce correctly across machines—particularly in SWE-Perf—scoring mechanisms substantially distort performance rankings, and 85.3% of valid tasks already have public submissions matching or exceeding reference performance. These findings expose critical issues including insufficient task validity, imbalanced weighting, and poor reproducibility, offering new insights for designing more robust and trustworthy evaluation frameworks.

Abstract

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.

Problem

• performance-optimization benchmarks
• coding agents
• runtime instability
• benchmark scoring rules
• leaderboard reliability

Innovation

• benchmark reliability
• code optimization
• reproducibility
• leaderboard bias
• performance evaluation