Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

2024-12-20

arXiv.org

AI Summary

This work addresses key limitations of Retrieval-Augmented Generation (RAG) for knowledge-intensive tasks—namely, high retrieval latency, document selection errors, and architectural complexity. We propose Cache-Augmented Generation (CAG), a novel paradigm that eliminates real-time retrieval by preloading a bounded knowledge corpus into the extended context window of large language models (e.g., 128K+ tokens) and leveraging parameter-based caching for zero-latency, zero-retrieval-error inference. To our knowledge, CAG is the first systematic architecture to replace retrieval entirely with caching, demonstrating that, under constrained knowledge settings, long-context LLMs can bypass retrieval modules altogether—yielding significantly simplified, more robust systems. Experiments across multiple benchmarks show CAG matches or exceeds RAG performance while eliminating the need for vector databases and dedicated retrievers, thereby drastically reducing system complexity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.

Problem

• Cache-Augmented Generation alternative to RAG
• Eliminates retrieval latency and errors
• Optimizes performance for constrained knowledge bases

Innovation

• Cache-augmented generation (CAG)
• Preloads relevant resources into LLM
• Eliminates retrieval latency and errors