RAG-Stack: Co-Optimizing RAG Serving Performance and Quality

2026-08-04

AI Summary

This work addresses the challenge of achieving an optimal trade-off between answer quality and service performance in modern Retrieval-Augmented Generation (RAG) systems. To this end, we propose RAG-Stack, a novel framework that enables, for the first time, joint quality–performance optimization across diverse RAG applications and deployment platforms. RAG-Stack integrates three key components: the RAG-PE iterative exploration algorithm, the RAG-IR workload abstraction, and the RAG-CM performance prediction model, which together efficiently navigate the configuration space and support Pareto front transferability. Experimental results demonstrate that, under the same number of optimization iterations, the Pareto fronts discovered by RAG-Stack cover a quality–performance space 52.5% to 153.2% larger than those achieved by existing methods.

Abstract

Retrieval-augmented generation (RAG), which augments large language model (LLM) generation with information retrieved from databases, has become a widely used approach for knowledge-intensive applications. Modern RAG systems, however, expose many configuration choices, such as retrieval indexes, model selections, and how models invoke retrieval. Each configuration yields a different trade-off between answer quality and serving performance, making it challenging to choose the optimal setting for a specific application deployment. We present RAG-Stack, a framework for efficiently discovering quality-performance Pareto frontiers across diverse RAG applications and serving systems. RAG-Stack consists of RAG-PE, an iterative design-space exploration algorithm that selects the next RAG configuration to evaluate; RAG-IR, a workload abstraction for diverse RAG algorithms; and RAG-CM, a performance model that predicts the optimal deployment and serving performance on the given hardware. Together, these components allow RAG-Stack to search the joint algorithm-system configuration space without deploying every candidate and to transfer an existing Pareto frontier to a new serving system. Given the same number of optimization iterations across diverse datasets, the Pareto frontiers found by RAG-Stack cover 52.5% to 153.2% more of the normalized quality-performance space than those found by state-of-the-art configuration-search methods evaluated over the same RAG design space.

Problem

• Retrieval-Augmented Generation
• Configuration Optimization
• Quality-Performance Trade-off
• RAG Systems
• Pareto Frontier

Innovation

• RAG-Stack
• Pareto frontier
• design-space exploration
• performance modeling
• retrieval-augmented generation