Quantization for Vector Search under Streaming Updates

2025-12-20

AI Summary

Existing data-dependent approximate nearest neighbor (ANN) quantization methods for streaming updates struggle with dynamic insertions/deletions, suffer from accuracy degradation, or incur prohibitive global rebuild costs. Method: We formally define dynamic consistency and a bounded disk-access model, and theoretically prove that static data-dependent quantization can be safely extended to the dynamic setting while preserving ANN accuracy guarantees. We propose an adaptive quantization mechanism guided by dynamic consistency constraints, a finite remote disk I/O optimization strategy, and a lightweight streaming incremental update framework. Contribution/Results: Evaluated on large-scale streaming vector retrieval tasks, our approach significantly outperforms state-of-the-art baselines, achieving an optimal trade-off among high retrieval accuracy, low update overhead, and rigorous theoretical guarantees.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Large-scale vector databases for approximate nearest neighbor (ANN) search typically store a quantized dataset in main memory for fast access, and full precision data on remote disk. State-of-the-art ANN quantization methods are highly data-dependent, rendering them unable to handle point insertions and deletions. This either leads to degraded search quality over time, or forces costly global rebuilds of the entire search index. In this paper, we formally study data-dependent quantization under streaming dataset updates. We formulate a computation model of limited remote disk access and define a dynamic consistency property that guarantees freshness under updates. We use it to obtain the following results: Theoretically, we prove that static data-dependent quantization can be made dynamic with bounded disk I/O per update while retaining formal accuracy guarantees for ANN search. Algorithmically, we develop a practical data-dependent quantization method which is provably dynamically consistent, adapting itself to the dataset as it evolves over time. Our experiments show that the method outperforms baselines in large-scale nearest neighbor search quantization under streaming updates.

Problem

• Dynamic quantization for streaming vector updates
• Maintaining search accuracy with dataset changes
• Reducing costly index rebuilds in ANN search

Innovation

• Dynamic data-dependent quantization adapting to streaming updates
• Bounded disk I/O per update with formal accuracy guarantees
• Provably consistent method outperforming baselines in ANN search