On Pretraining for Project-Level Code Completion

2025-10-15

AI Summary

This study investigates how warehouse-level code processing strategies affect the in-context learning capability of OpenCoder for project-level code completion. Methodologically, we pretrain a 1.5B-parameter OpenCoder on only 1 billion tokens of warehouse-scale code, extend its context window to 16,384 tokens, and systematically optimize the RoPE scaling parameter. Our key findings are: (i) RoPE scaling optimization is the primary driver of performance gains; and (ii) lightweight file-level training retains high effectiveness for short sequences while substantially reducing computational and data requirements. Empirical evaluation on the Long Code Arena benchmark shows that our approach achieves performance comparable to models trained on tens of billions of tokens. This work establishes a new paradigm for efficient, low-resource project-level code completion—demonstrating that carefully engineered architectural and positional-encoding adjustments can compensate for limited pretraining scale without sacrificing contextual reasoning capability.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.

Problem

• Investigating repository-processing strategies for code completion
• Extending context window to enhance code generation accuracy
• Evaluating simpler training approaches for resource-constrained environments

Innovation

• Extended context window to 16,384 tokens
• Applied rotary positional embedding scaling parameter
• Used simpler file-level training approach