Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations

2026-07-02

AI Summary

This work investigates whether large language model–driven coding agents can automatically repair missing peephole optimizations in compilers. To this end, we introduce PeepholeBench, the first structured benchmark for compiler optimization repair, constructed from real-world missing optimization cases in LLVM’s InstCombine pass and requiring agents to generate correct and efficient patches using only contextual information. Our experiments demonstrate that even state-of-the-art agents struggle to simultaneously satisfy semantic correctness and performance gains. Common failure modes include insufficient transformation scope and misuse of LLVM-specific mechanisms, revealing a fundamental trade-off between rigor and optimization efficacy in current approaches.

Abstract

Coding agents built on large language models are now capable of patching sizable real-world codebases, yet whether they can develop compiler optimizations remains an open question. To study this question, we introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations reported against LLVM's InstCombine pass. Since missed peephole optimizations are typically fixed with small, localized patches, they offer a well-scoped but demanding testbed for coding agents: a correct fix demands rigorous reasoning about program semantics along with familiarity with optimizer-specific conventions. PeepholeBench derives its tasks from 21 resolved LLVM issues and 19 merged pull requests (PRs), supplies agents with only the issue context that existed before each fix, and assesses the resulting patches for both correctness and profitability. With PeepholeBench, we benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass, measuring their patches against the corresponding human-written fixes. We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once. The dominant failure modes are overly narrow transformations and misuse of LLVM-specific mechanisms, errors that existing test suites rarely expose. Together, these results establish PeepholeBench as a realistic and challenging benchmark for coding agents, and suggest future directions for building agents that can more dependably assist compiler optimization development.

Problem

• compiler optimizations
• peephole optimizations
• coding agents
• LLVM
• large language models

Innovation

• PeepholeBench
• LLVM
• compiler optimizations
• coding agents
• InstCombine