HARMONY: A Scalable Distributed Vector Database for High-Throughput Approximate Nearest Neighbor Search

2025-06-17

AI Summary

Distributed approximate nearest neighbor search (ANNS) for billion-scale high-dimensional vectors faces critical challenges: limited per-node memory, load imbalance, and high inter-node communication overhead. To address these, this paper proposes a high-throughput distributed vector database architecture. Our method introduces two core innovations: (1) a novel multi-granularity partitioning strategy that jointly partitions both dimensions and vectors, enabling coordinated balancing of data distribution and computational load; and (2) a distance-monotonicity–based early-termination pruning mechanism that significantly reduces cross-node communication while preserving retrieval accuracy. The system supports parallel index construction and query scheduling. Evaluated on a four-node cluster, it achieves 4.63× higher average throughput than state-of-the-art systems and improves performance by 58% under skewed workloads, effectively overcoming memory and scalability bottlenecks in large-scale ANNS.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Distributed Machine Learning & Federated Learning

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Approximate Nearest Neighbor Search (ANNS) is essential for various data-intensive applications, including recommendation systems, image retrieval, and machine learning. Scaling ANNS to handle billions of high-dimensional vectors on a single machine presents significant challenges in memory capacity and processing efficiency. To address these challenges, distributed vector databases leverage multiple nodes for the parallel storage and processing of vectors. However, existing solutions often suffer from load imbalance and high communication overhead, primarily due to traditional partition strategies that fail to effectively distribute the workload. In this paper, we introduce Harmony, a distributed ANNS system that employs a novel multi-granularity partition strategy, combining dimension-based and vector-based partition. This strategy ensures a balanced distribution of computational load across all nodes while effectively minimizing communication costs. Furthermore, Harmony incorporates an early-stop pruning mechanism that leverages the monotonicity of distance computations in dimension-based partition, resulting in significant reductions in both computational and communication overhead. We conducted extensive experiments on diverse real-world datasets, demonstrating that Harmony outperforms leading distributed vector databases, achieving 4.63 times throughput on average in four nodes and 58% performance improvement over traditional distribution for skewed workloads.

Problem

• Scalable distributed vector database for high-throughput ANNS
• Addressing load imbalance in distributed vector databases
• Minimizing communication overhead in nearest neighbor search

Innovation

• Multi-granularity partition strategy for load balance
• Early-stop pruning reduces computation and communication
• Dimension-based and vector-based partition combination