BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

2025-09-26

AI Summary

Existing open-source software (OSS) auto-compilation approaches rely either on handcrafted rules or evaluate only highly starred repositories, failing to address real-world challenges such as missing documentation, implicit dependencies, and required source-code patches. To bridge this gap, we propose BuildBench—the first comprehensive benchmark explicitly designed to reflect practical compilation difficulty—featuring diverse, low-resource, and highly heterogeneous OSS projects. We further introduce OSS-Build-Agent, an intelligent agent framework integrating enhanced build-instruction retrieval, dependency inference, and context-aware code modification, enabling dynamic environment configuration and automated source-code patching. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods on BuildBench, exhibiting strong robustness and generalization across challenging scenarios—including undocumented builds, complex dependency resolution, and patch-dependent compilation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Teamwork
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Automatically compiling open-source software (OSS) projects is a vital, labor-intensive, and complex task, which makes it a good challenge for LLM Agents. Existing methods rely on manually curated rules and workflows, which cannot adapt to OSS that requires customized configuration or environment setup. Recent attempts using Large Language Models (LLMs) used selective evaluation on a subset of highly rated OSS, a practice that underestimates the realistic challenges of OSS compilation. In practice, compilation instructions are often absent, dependencies are undocumented, and successful builds may even require patching source files or modifying build scripts. We propose a more challenging and realistic benchmark, BUILD-BENCH, comprising OSS that are more diverse in quality, scale, and characteristics. Furthermore, we propose a strong baseline LLM-based agent, OSS-BUILD-AGENT, an effective system with enhanced build instruction retrieval module that achieves state-of-the-art performance on BUILD-BENCH and is adaptable to heterogeneous OSS characteristics. We also provide detailed analysis regarding different compilation method design choices and their influence to the whole task, offering insights to guide future advances. We believe performance on BUILD-BENCH can faithfully reflect an agent's ability to tackle compilation as a complex software engineering tasks, and, as such, our benchmark will spur innovation with a significant impact on downstream applications in the fields of software development and software security.

Problem

• Automating compilation of diverse open-source software projects
• Addressing undocumented dependencies and missing build instructions
• Overcoming limitations of manual rules and selective evaluation methods

Innovation

• Proposes BUILD-BENCH benchmark for diverse OSS compilation
• Introduces OSS-BUILD-AGENT with enhanced instruction retrieval module
• Provides adaptable baseline agent for heterogeneous OSS characteristics