NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents

2025-12-14

AI Summary

Existing benchmarks for coding agents inadequately evaluate long-horizon software system construction—specifically, the ability to maintain coherent planning and execution over hours to days in realistic repository-scale development. Method: We introduce the first evaluation benchmark for *long-horizon repository generation*, requiring models to autonomously design architecture, manage dependencies, implement multiple modules, and deliver an installable Python package—starting solely from natural language requirements and an empty workspace. We formalize and quantify this capability, identify critical failure modes (e.g., premature termination, global incoherence, cross-file dependency fragility), and build an end-to-end executable test environment grounded in real-world Python ecosystems—featuring installation validation, multi-file consistency checking, coverage-driven assessment, and step-by-step execution tracing. Results: Experiments show that state-of-the-art models achieve <40% average test pass rate, underscoring long-horizon reasoning as a fundamental bottleneck for autonomous coding agents.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Multiagent Planning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.

Problem

• Evaluates long-horizon repository generation in coding agents
• Assesses autonomous design and multi-module implementation from requirements
• Identifies failure modes like coherence loss and inadequate planning

Innovation

• Benchmark for long-horizon repository generation evaluation
• Agents autonomously design architecture from natural language requirements
• Reveals failure modes like premature termination and coherence loss