YABLoCo: Yet Another Benchmark for Long Context Code Generation

2025-05-07

AI Summary

Existing LLM-based code generation evaluation is largely confined to small-scale codebases (≤1K LoC), failing to reflect model capabilities on million-line industrial projects. To address this gap, we propose the first benchmark for function-body generation in ultra-long-context C/C++ code (200K–2M LoC), constructed from real-world large open-source repositories and supporting multi-level dependency context, call graphs, and structured metadata. Our method introduces: (i) the first evaluation framework specifically designed for large-scale C/C++ codebases; (ii) a million-line-scale, realistic test suite comprising 215 high-quality functions; and (iii) an extensible evaluation pipeline integrated with an interactive, visualization-enabled analysis toolkit. This work substantially advances the measurability, comparability, and interpretability of LLMs on industrial-grade code generation tasks requiring long-context understanding and complex inter-procedural dependencies.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models demonstrate the ability to solve various programming tasks, including code generation. Typically, the performance of LLMs is measured on benchmarks with small or medium-sized context windows of thousands of lines of code. At the same time, in real-world software projects, repositories can span up to millions of LoC. This paper closes this gap by contributing to the long context code generation benchmark (YABLoCo). The benchmark featured a test set of 215 functions selected from four large repositories with thousands of functions. The dataset contained metadata of functions, contexts of the functions with different levels of dependencies, docstrings, functions bodies, and call graphs for each repository. This paper presents three key aspects of the contribution. First, the benchmark aims at function body generation in large repositories in C and C++, two languages not covered by previous benchmarks. Second, the benchmark contains large repositories from 200K to 2,000K LoC. Third, we contribute a scalable evaluation pipeline for efficient computing of the target metrics and a tool for visual analysis of generated code. Overall, these three aspects allow for evaluating code generation in large repositories in C and C++.

Problem

• Evaluating code generation in large C/C++ repositories
• Addressing lack of benchmarks for million-line codebases
• Providing scalable metrics for long-context function generation

Innovation

• Function body generation in large C/C++ repositories
• Benchmark with repositories from 200K to 2M LoC
• Scalable evaluation pipeline and visual analysis tool