PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

2026-05-13

AI Summary

While current large language models can generate functionally correct code, they exhibit limited capability in system-level high-performance optimization and lack appropriate evaluation benchmarks. This work introduces the first executable benchmark specifically designed for assessing system-level high-performance code optimization, encompassing critical tasks such as hardware-aware optimizations, parallelization strategies, and performance bottleneck identification. The benchmark features multi-language implementations, executable validation, and runtime performance metrics to jointly evaluate both correctness and efficiency, alongside baseline and expert-optimized solutions for comparative analysis. Experimental results demonstrate that state-of-the-art models still fall significantly short of expert-level performance in tasks involving GPU operations and parallel computing, and further reveal limitations in cross-language robustness and consistency of optimization efficacy.

Abstract

Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution. This allows us to evaluate both correctness and runtime-oriented efficiency. Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations. Current models also show weaknesses in cross-language robustness and in consistently reaching expert-level efficiency. These results suggest that performance-aware evaluation are still needed. LLMs should move beyond generating merely correct code toward producing efficient systems software. We submit the benchmark data, evaluation infrastructure, and complete logs of all LLMs-generated code at https://anonymous.4open.science/r/perfcodebench-7CDE.

Problem

• code optimization
• large language models
• system-level performance
• benchmarking
• hardware-aware optimization

Innovation

• code optimization
• large language models
• system-level performance
• hardware-awareness
• executable benchmark