QVCache: A Query-Aware Vector Cache

2026-02-02

AI Summary

This work addresses the inefficiency of existing vector databases under high-recall and low-latency requirements, where memory and I/O bottlenecks hinder effective query-level caching. The authors propose a backend-agnostic, semantic-aware caching system that introduces, for the first time, a vector cache layer capable of handling semantically redundant queries. By dynamically adjusting region-wise distance thresholds through online learning, the system achieves high cache hit rates within a bounded memory footprint of just a few megabytes. Deployed as a plug-in component, it substantially reduces end-to-end query latency by 40–1000×, with cache-hit responses under 1 millisecond, while preserving recall performance comparable to that of the underlying approximate nearest neighbor (ANN) search backend.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Hardware-aware ML

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vector databases have become a cornerstone of modern information retrieval, powering applications in recommendation, search, and retrieval-augmented generation (RAG) pipelines. However, scaling approximate nearest neighbor (ANN) search to high recall under strict latency SLOs remains fundamentally constrained by memory capacity and I/O bandwidth. Disk-based vector search systems suffer severe latency degradation at high accuracy, while fully in-memory solutions incur prohibitive memory costs at billion-scale. Despite the central role of caching in traditional databases, vector search lacks a general query-level caching layer capable of amortizing repeated query work. We present QVCache, the first backend-agnostic, query-level caching system for ANN search with bounded memory footprint. QVCache exploits semantic query repetition by performing similarity-aware caching rather than exact-match lookup. It dynamically learns region-specific distance thresholds using an online learning algorithm, enabling recall-preserving cache hits while bounding lookup latency and memory usage independently of dataset size. QVCache operates as a drop-in layer for existing vector databases. It maintains a megabyte-scale memory footprint and achieves sub-millisecond cache-hit latency, reducing end-to-end query latency by up to 40-1000x when integrated with existing ANN systems. For workloads exhibiting temporal-semantic locality, QVCache substantially reduces latency while preserving recall comparable to the underlying ANN backend, establishing it as a missing but essential caching layer for scalable vector search.

Problem

• vector search
• approximate nearest neighbor
• query caching
• latency
• scalability

Innovation

• query-aware caching
• semantic similarity
• approximate nearest neighbor
• online learning
• vector database