CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems

2025-05-02

AI Summary

In Retrieval-Augmented Generation (RAG), disk-based vector retrieval suffers from non-uniform cluster access patterns induced by query semantic similarity, while existing approaches neglect cross-query locality effects—leading to low cache efficiency and high tail latency. This paper proposes a context-aware dynamic query grouping mechanism: it clusters semantically similar queries based on shared cluster access patterns to enable fine-grained scheduling; and introduces opportunistic cluster-level asynchronous prefetching coupled with I/O-aware cache management. Its core innovation lies in the first joint modeling of query semantic clustering and cluster-access locality to co-optimize cache locality and I/O efficiency. Experiments show up to 51.55% reduction in 99th-percentile tail latency, consistently higher cache hit rates than baselines, and significant improvements in retrieval throughput and response stability.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Distributed Search
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Modern embedding models capture both semantic and syntactic structures of queries, often mapping different queries to similar regions in vector space. This results in non-uniform cluster access patterns in disk-based vector search systems, particularly in Retrieval Augmented Generation (RAG) framework. While existing approaches optimize individual queries, they overlook the impact of cluster access patterns, failing to account for the locality effects of queries that access similar clusters. This oversight reduces cache efficiency and increases search latency due to excessive disk I/O. To address this, we introduce CaGR-RAG, a context-aware query grouping mechanism that organizes queries based on shared cluster access patterns. Additionally, it incorporates opportunistic cluster prefetching to minimize cache misses during transitions between query groups, further optimizing retrieval performance. Experimental results show that CaGR-RAG reduces 99th percentile tail latency by up to 51.55% while consistently maintaining a higher cache hit ratio than the baseline.

Problem

• Optimizes cluster access patterns in disk-based vector search
• Reduces cache misses and disk I/O in RAG systems
• Improves retrieval latency and cache efficiency via query grouping

Innovation

• Context-aware query grouping for disk-based search
• Opportunistic cluster prefetching to minimize cache misses
• Optimizes retrieval performance by shared cluster access