LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

2026-07-01

AI Summary

This work addresses the significant challenge of repairing bugs in the complex LLVM compiler infrastructure, where the effectiveness of large language models (LLMs) remains unclear. To advance research in this domain, the authors introduce LLVM-Bench, a large-scale benchmark for LLVM repair, and LLVM-Gym, an automated evaluation platform. They further propose LLVM-Ens, a lightweight ensemble approach that integrates patches generated by multiple LLMs and filters out invalid candidates. Experimental results demonstrate that LLVM-Ens achieves a 21.99% problem-solving rate on LLVM-Bench, substantially outperforming any single model. The study also identifies patch invalidity and build failures as the predominant failure modes, thereby validating the efficacy of integrating complementary techniques in automated program repair for complex compiler systems.

Abstract

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.

Problem

• LLVM
• large language models
• compiler issue resolution
• benchmarking
• software maintenance

Innovation

• LLVM-Bench
• LLVM-Gym
• large language models
• compiler issue resolution
• ensemble approach