CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

2026-06-17

AI Summary

This work addresses the inefficiency in Retrieval-Augmented Generation (RAG) systems where varying evidence orderings prevent reuse of prefix caches, thereby increasing prefill overhead and first-token latency. The authors propose a lightweight prompt-layer approach that, without altering retrieval results or the underlying inference engine, constructs a prefix tree over recently used evidence sequences and applies a greedy strategy to reorder incoming evidence, maximizing cache reuse. This method introduces, for the first time in RAG, a cache-aware evidence reordering mechanism that operates solely at the scheduling layer yet yields substantial efficiency gains. Experimental results across three vLLM configurations demonstrate a 20–33% reduction in median first-token latency while preserving answer quality, achieving 97.5% of the cache benefit attainable under an ideal ordering.

Abstract

Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference.

Problem

• Retrieval-Augmented Generation
• prefix caching
• evidence ordering
• cache reuse
• inference efficiency

Innovation

• cache-aware ordering
• prefix caching
• Retrieval-Augmented Generation
• evidence reordering
• time-to-first-token