CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System

2025-05-07

AI Summary

To address the challenges of manual compilation—such as labor intensiveness, error proneness, and difficulty in retrieving build commands and diagnosing failures—amplified by the growing scale of open-source repositories, this paper proposes the first repository-level (repo-level) automated compilation framework based on large language model (LLM) agents. Methodologically, it introduces a workflow-driven agent strategy integrating multiple tools—including code search, log analysis, and dependency resolution—and incorporates a compilation-friendly baseline fusion mechanism. We further construct CompileAgentBench, the first publicly available repo-level compilation benchmark. Experiments show that our framework achieves 10–71% higher compilation success rates on CompileAgentBench than state-of-the-art approaches; ablation studies confirm the workflow strategy as optimal, and scalability analysis demonstrates robust extensibility. Key contributions include: (1) the first LLM-based agent framework for repo-level compilation, (2) a workflow-driven decision-making paradigm, (3) CompileAgentBench, and (4) a multi-tool orchestration mechanism tailored for compilation tasks.

Technology Category

• Multiagent Systems: Agent Communication
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With open-source projects growing in size and complexity, manual compilation becomes tedious and error-prone, highlighting the need for automation to improve efficiency and accuracy. However, the complexity of compilation instruction search and error resolution makes automatic compilation challenging. Inspired by the success of LLM-based agents in various fields, we propose CompileAgent, the first LLM-based agent framework dedicated to repo-level compilation. CompileAgent integrates five tools and a flow-based agent strategy, enabling interaction with software artifacts for compilation instruction search and error resolution. To measure the effectiveness of our method, we design a public repo-level benchmark CompileAgentBench, and we also design two baselines for comparison by combining two compilation-friendly schemes. The performance on this benchmark shows that our method significantly improves the compilation success rate, ranging from 10% to 71%. Meanwhile, we evaluate the performance of CompileAgent under different agent strategies and verify the effectiveness of the flow-based strategy. Additionally, we emphasize the scalability of CompileAgent, further expanding its application prospects.

Problem

• Automating repo-level compilation to improve efficiency and accuracy
• Resolving compilation errors using LLM-based agent framework
• Enhancing compilation success rates with tool-integrated flow strategy

Innovation

• LLM-based agent framework for repo-level compilation
• Integrates five tools and flow-based agent strategy
• Significantly improves compilation success rate by 10-71%