What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond

2025-03-26

AI Summary

This study systematically investigates the effectiveness of different information sources—contextual code, APIs, and similar code snippets—in retrieval-augmented code generation, revealing that contextual code and API semantic information substantially improve performance, whereas similar code snippets often introduce noise. To address this, we propose AllianceCoder: the first framework integrating chain-of-thought (CoT) prompting with semantics-aware API retrieval via description matching, enabling context-sensitive and precise API identification; and a repository-level RAG architecture supporting holistic codebase understanding. Evaluated on CoderEval and RepoExec benchmarks, AllianceCoder achieves up to 20% absolute improvement in Pass@1 over state-of-the-art methods. Our core contributions are: (1) the first empirical demonstration that contextual code and APIs constitute the optimal retrieval targets, and (2) a novel retrieval paradigm jointly optimizing semantic fidelity and contextual relevance.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Repository-level code generation remains challenging due to complex code dependencies and the limitations of large language models (LLMs) in processing long contexts. While retrieval-augmented generation (RAG) frameworks are widely adopted, the effectiveness of different retrieved information sources-contextual code, APIs, and similar snippets-has not been rigorously analyzed. Through an empirical study on two benchmarks, we demonstrate that in-context code and potential API information significantly enhance LLM performance, whereas retrieved similar code often introduces noise, degrading results by up to 15%. Based on the preliminary results, we propose AllianceCoder, a novel context-integrated method that employs chain-of-thought prompting to decompose user queries into implementation steps and retrieves APIs via semantic description matching. Through extensive experiments on CoderEval and RepoExec, AllianceCoder achieves state-of-the-art performance, improving Pass@1 by up to 20% over existing approaches.

Problem

• Analyzing effectiveness of retrieved information sources for code generation
• Addressing noise from similar code in retrieval-augmented generation frameworks
• Proposing a novel method to enhance LLM performance in repository-level coding

Innovation

• Uses contextual code and API information
• Employs chain-of-thought prompting
• Retrieves APIs via semantic matching