Rethinking Code Performance Benchmarks for LLMs

2026-07-08

AI Summary

Existing code performance benchmarks struggle to effectively evaluate the actual runtime performance gains of programs generated by large language models. This work systematically analyzes mainstream benchmarks and finds that most purportedly “high-performance” implementations lack statistically significant speedups. To address this limitation, we propose a multi-agent collaborative framework based on a three-agent architecture—comprising generation, diagnosis, and repair agents—integrated with statistical hypothesis testing to automatically produce functionally correct and performance-sensitive test cases. Applied to 1,345 tasks where existing benchmarks failed to detect performance differences, our method successfully uncovered statistically significant speedups for DeepSeek-v3.1 and GPT-4o in 24.01% and 25.43% of cases, respectively, substantially outperforming current test generation approaches.

Abstract

Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method.

Problem

• code performance benchmarks
• large language models
• runtime evaluation
• performance testing
• benchmark limitations

Innovation

• performance benchmarking
• LLM-based testing
• multi-agent framework
• statistical significance
• code optimization