PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization

2026-07-08

AI Summary

Existing evaluations of code-generating agents primarily emphasize functional correctness while overlooking their capacity for performance optimization in real-world scenarios. This work proposes the first end-to-end benchmark tailored to the full performance engineering lifecycle, requiring agents to achieve reproducible performance gains through profiling, diagnosis, code modification, and validation—all while preserving functional correctness. The framework introduces several innovations, including hidden correctness tests, verifiable speedup metrics, and trajectory auditing, integrated with program profiling, cross-layer bottleneck diagnosis, large model–agent collaboration, and an optimization-summary handoff strategy. Evaluation across seven long-horizon tasks and seven agent stacks reveals that optimization efficacy is highly workload-dependent, with no single dominant approach; relying solely on raw speedup ratios can lead to misleading conclusions, necessitating a holistic assessment that jointly considers correctness and reproducibility.

Abstract

Coding-agent benchmarks have largely measured whether agents can produce functionally correct patches, but production software also demands measurable speedups on real execution targets. Performance optimization is a distinct agentic task: agents must profile executions, diagnose cross-layer bottlenecks, edit code without breaking correctness, and verify that gains are reproducible rather than measurement artifacts. We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop. Each task provides a correct but deliberately suboptimal codebase and asks the agent to improve a target performance metric; scoring requires hidden correctness tests, verified-speedup measurement, and trajectory-level audit. We evaluate 7 agent stacks with different LLMs and agent frameworks on 7 long-horizon optimization tasks. The results show that optimization performance is workload-dependent rather than determined by model identity alone: no single stack dominates, and changing the agent framework can materially change the same LLM's per-task speedup profile. We further find that raw speedup is unsafe as a benchmark score, since some large gains arise from benchmark-specific shortcut exploitation; an exploratory relay pilot suggests that restarting from an externalized optimization summary can recover additional headroom after an initial session stops. The benchmark and our evaluation are available at: https://anonymous.4open.science/r/Dataset-D3CC.

Problem

• performance optimization
• coding agents
• benchmark
• software performance
• agentic tasks

Innovation

• performance optimization
• coding agents
• benchmarking
• agent frameworks
• speedup verification