🤖 AI Summary
Existing similarity search methods are largely confined to metric spaces, rendering them inadequate for non-metric, topologically heterogeneous, sparse, or skewed data distributions. This paper introduces the first distributed approximate similarity search framework supporting arbitrary distance functions. Our approach addresses these challenges through three core contributions: (1) a multi-level clustering index structure that enhances robustness against outliers and data imbalance; (2) a clustering-driven embedding mechanism for non-metric distances, coupled with a compatibility layer ensuring consistent neighborhood semantics; and (3) a distributed nearest-neighbor retrieval architecture integrating scalable hashing with dynamic load balancing. Evaluated on diverse topological benchmark datasets, our framework achieves an average 2.3× higher query throughput and a 37% reduction in recall error compared to state-of-the-art ANN methods, significantly improving both efficiency and accuracy for large-scale similarity search in non-metric spaces.
📝 Abstract
Recent studies have explored alternative distance measures for similarity search in spaces with diverse topologies, emphasizing the importance of selecting an appropriate distance function to improve the performance of k-Nearest Neighbour search algorithms. However, a critical gap remains in accommodating such diverse similarity measures, as most existing methods for exact or approximate similarity search are explicitly designed for metric spaces. To address this need, we propose PDASC (Parametrizable Distributed Approximate Similarity Search with Clustering), a novel Approximate Nearest Neighbour search algorithm. PDASC combines an innovative multilevel indexing structure particularly adept at managing outliers, highly imbalanced datasets, and sparse data distributions, with the flexibility to support arbitrary distance functions achieved through the integration of clustering algorithms that inherently accommodate them. Experimental results show that PDASC constitutes a reliable ANN search method, suitable for operating in distributed data environments and for handling datasets defined in different topologies, where the selection of the most appropriate distance function is often non-trivial.