Scalable Distributed Vector Search via Accuracy Preserving Index Construction

2025-12-19

AI Summary

Scalable distributed approximate nearest neighbor (ANN) search over billion-scale vector datasets demands careful trade-offs among accuracy, latency, and throughput. This paper proposes a scalable multi-level distributed indexing framework: first, a load-aware fine-grained partitioning strategy is designed to minimize cross-node read overhead; second, a precision-preserving recursive hierarchical index structure is introduced to ensure predictable search cost and stable recall. Evaluated on a 46-node cluster, the system supports 8 billion vectors with throughput up to 9.64× higher than the current state-of-the-art—without any accuracy degradation—and demonstrates superior linear scalability. The core innovation lies in a synergistic co-design mechanism that jointly optimizes partitioning efficiency and indexing accuracy, enabling both high throughput and guaranteed retrieval quality at billion-scale.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Scalability of ML Systems

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Scaling Approximate Nearest Neighbor Search (ANNS) to billions of vectors requires distributed indexes that balance accuracy, latency, and throughput. Yet existing index designs struggle with this tradeoff. This paper presents SPIRE, a scalable vector index based on two design decisions. First, it identifies a balanced partition granularity that avoids read-cost explosion. Second, it introduces an accuracy-preserving recursive construction that builds a multi-level index with predictable search cost and stable accuracy. In experiments with up to 8 billion vectors across 46 nodes, SPIRE achieves high scalability and up to 9.64X higher throughput than state-of-the-art systems.

Problem

• Designs scalable distributed vector indexes for billion-scale ANNS
• Balances accuracy, latency, and throughput in distributed search
• Avoids read-cost explosion with balanced partition granularity

Innovation

• Balanced partition granularity prevents read-cost explosion
• Accuracy-preserving recursive construction ensures stable accuracy
• Multi-level index design enables predictable search cost