Graph-Based Algorithms for Diverse Similarity Search

2025-02-18

AI Summary

This paper addresses the efficiency degradation caused by diversity constraints in approximate nearest neighbor (ANN) search. We propose the first end-to-end graph neural indexing framework explicitly designed for diversity-aware retrieval. Unlike conventional two-stage paradigms (coarse retrieval followed by post-hoc diversification), our method integrates diversity optimization directly into graph construction and local traversal, guiding dynamic pruning and path selection in NSG- and HNSW-style graphs via a greedy diversity criterion. We theoretically prove that, under low intrinsic dimensionality, the query time complexity is (O(k log Delta))—matching the optimal bound for unconstrained ANN. Experiments demonstrate that our approach achieves high diversity (as measured by MaxMin and Intra-List Distance metrics) while maintaining latency close to single-point ANN search, significantly outperforming state-of-the-art two-stage baselines.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Nearest neighbor search is a fundamental data structure problem with many applications in machine learning, computer vision, recommendation systems and other fields. Although the main objective of the data structure is to quickly report data points that are closest to a given query, it has long been noted (Carbonell and Goldstein, 1998) that without additional constraints the reported answers can be redundant and/or duplicative. This issue is typically addressed in two stages: in the first stage, the algorithm retrieves a (large) number $r$ of points closest to the query, while in the second stage, the $r$ points are post-processed and a small subset is selected to maximize the desired diversity objective. Although popular, this method suffers from a fundamental efficiency bottleneck, as the set of points retrieved in the first stage often needs to be much larger than the final output. In this paper we present provably efficient algorithms for approximate nearest neighbor search with diversity constraints that bypass this two stage process. Our algorithms are based on popular graph-based methods, which allows us to"piggy-back"on the existing efficient implementations. These are the first graph-based algorithms for nearest neighbor search with diversity constraints. For data sets with low intrinsic dimension, our data structures report a diverse set of $k$ points approximately closest to the query, in time that only depends on $k$ and $log Delta$, where $Delta$ is the ratio of the diameter to the closest pair distance in the data set. This bound is qualitatively similar to the best known bounds for standard (non-diverse) graph-based algorithms. Our experiments show that the search time of our algorithms is substantially lower than that using the standard two-stage approach.

Problem

• Efficient diverse nearest neighbor search algorithms.
• Graph-based methods for diversity constraints.
• Reduced search time compared to two-stage methods.

Innovation

• Graph-based diverse similarity search
• Bypasses two-stage process
• Efficiency depends on k and logΔ