Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Tasks

2026-05-14

AI Summary

This study addresses the lack of systematic guidance for component-level configuration in retrieval-augmented generation (RAG) systems applied to software engineering tasks, a gap that has led practitioners to rely on costly trial-and-error approaches. Through a large-scale empirical evaluation, the work systematically assesses the impact of four core RAG modules—query processing, retrieval models, context refinement, and generators—across code generation, summarization, and repair tasks. The experiments span four query processing techniques, seven retrieval models, four refinement methods, and six generators, yielding 21 distinct configurations. Notably, the study reveals that retrieval components—particularly the choice of retrieval algorithm—often exert a greater influence on overall performance than the generator itself, with BM25 demonstrating consistently robust results across multiple tasks. These findings provide a data-driven prioritization framework for optimizing RAG systems in software engineering, substantially reducing configuration costs.

Abstract

While Retrieval-Augmented Generation (RAG) is increasingly adopted to ground Large Language Models (LLMs) in software artifacts, the optimal configuration of its components remains an open question for software engineering (SE) tasks. The lack of systematic guidance forces practitioners into costly, ad-hoc experimentation. This paper presents a comprehensive, component-wise empirical study that dissects the RAG pipeline, evaluating over 21 distinct models and methods. Our study systematically isolates and evaluates 4 query processing techniques, 7 retrieval models spanning sparse, dense, and hybrid paradigms, 4 context refinement methods, and 6 distinct generators. We test these components on a suite of 3 core SE tasks: code generation, summarization, and repair. Our empirical findings reveal a crucial insight: the retriever-side components, particularly the choice of the retrieval algorithm, often exert a more significant influence on final system performance than the selection of the generator model. Strikingly, the classic lexical retriever BM25 demonstrates exceptionally robust performance across diverse tasks. Our analysis provides a practical, data-driven roadmap for researchers and practitioners, offering clear guidance on prioritizing optimization efforts when constructing effective RAG systems for software engineering contexts.

Problem

• Retrieval-Augmented Generation
• Software Engineering
• Component Configuration
• Empirical Study
• RAG Optimization

Innovation

• Retrieval-Augmented Generation
• empirical study
• software engineering tasks
• BM25
• component-wise analysis