RAG-Stack: Co-Optimizing RAG Quality and Performance From the Vector Database Perspective

2025-10-23

AI Summary

To address the challenge of coupled algorithmic (model/database) and systems-level (software/hardware) design in retrieval-augmented generation (RAG), which hinders joint optimization of quality and performance, this paper proposes RAG-Stack. Our framework introduces three novel pillars: (1) RAG-IR—a middleware abstraction layer based on intermediate representations that decouples algorithmic logic from system implementation; (2) RAG-CM—a fine-grained, end-to-end cost model unifying vector retrieval and LLM inference; and (3) RAG-PE—an efficient configuration exploration algorithm leveraging query plan search. RAG-Stack is the first framework to enable cross-layer co-optimization across the entire RAG pipeline. It achieves significant improvements in throughput and latency while preserving generation quality, and supports scalable, configurable production deployment.

Technology Category

• Search and Optimization: Algorithm Configuration
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-augmented generation (RAG) has emerged as one of the most prominent applications of vector databases. By integrating documents retrieved from a database into the prompt of a large language model (LLM), RAG enables more reliable and informative content generation. While there has been extensive research on vector databases, many open research problems remain once they are considered in the wider context of end-to-end RAG pipelines. One practical yet challenging problem is how to jointly optimize both system performance and generation quality in RAG, which is significantly more complex than it appears due to the numerous knobs on both the algorithmic side (spanning models and databases) and the systems side (from software to hardware). In this paper, we present RAG-Stack, a three-pillar blueprint for quality-performance co-optimization in RAG systems. RAG-Stack comprises: (1) RAG-IR, an intermediate representation that serves as an abstraction layer to decouple quality and performance aspects; (2) RAG-CM, a cost model for estimating system performance given an RAG-IR; and (3) RAG-PE, a plan exploration algorithm that searches for high-quality, high-performance RAG configurations. We believe this three-pillar blueprint will become the de facto paradigm for RAG quality-performance co-optimization in the years to come.

Problem

• Jointly optimizing RAG system performance and generation quality
• Addressing complexity from algorithmic and systems configuration knobs
• Developing a blueprint for quality-performance co-optimization in RAG systems

Innovation

• Intermediate representation decouples quality and performance
• Cost model estimates system performance for configurations
• Plan exploration algorithm searches optimal RAG configurations