From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation

2026-01-19

AI Summary

This work addresses the computational inefficiency and prolonged time-to-first-token (TTFT) in retrieval-augmented generation (RAG) caused by lengthy prompts incorporating external knowledge, as well as the degradation in generation quality from existing key-value (KV) cache reuse methods that neglect cross-chunk context. To overcome these limitations, the authors propose FusionRAG, a novel framework that embeds cross-chunk contextual information into individual text chunks during an offline phase and, at inference time, selectively recomputes KV caches only for attention-critical tokens—comprising less than 15% of the input. By integrating cross-chunk semantics into KV cache reuse for the first time, FusionRAG simultaneously preserves generation quality and enhances inference efficiency, achieving up to a 70% relative improvement in normalized F1 score over baselines while reducing TTFT by 2.66–9.39× under a recompute ratio below 15%.

Technology Category

• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token (TTFT). To mitigate this issue, existing solutions aim to reuse the preprocessed KV cache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KV cache reuse largely unfulfilled. The challenge lies in how to reuse the precomputed KV cache of chunks while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KV cache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized F1 scores than baselines and reduces TTFT by 2.66x-9.39x compared to Full Attention.

Problem

• Retrieval-Augmented Generation
• KV cache reuse
• generation quality
• LLM inference
• contextual information

Innovation

• FusionRAG
• KV cache reuse
• Retrieval-Augmented Generation
• efficient LLM inference
• context-aware preprocessing