Filtered Approximate Nearest Neighbor Search: A Unified Benchmark and Systematic Experimental Study [Experiment, Analysis & Benchmark]

2025-09-09

AI Summary

Existing FANNS algorithms lack a systematic survey; their parameters are tightly coupled and highly data-dependent, leading to biased cross-algorithm comparisons, while critical factors—such as data characteristics and query difficulty—remain unanalyzed. Method: We propose the first unified taxonomy and fair evaluation framework for label-constrained approximate nearest neighbor search (LC-ANN), decoupling algorithmic parameters to enable reproducible, cross-method comparisons. Our framework systematically examines three paradigms—filter-then-search, search-then-filter, and hybrid-search—integrating standardized filtering and adaptive tuning strategies. We further develop an open-source benchmark platform featuring diverse real-world datasets. Contribution/Results: Experiments reveal, for the first time, how algorithm performance varies with data distribution and query workload. The framework significantly improves reproducibility and provides actionable insights for practical deployment.

Technology Category

• Search and Optimization: Algorithm Configuration
• Constraint Satisfaction and Optimization: Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

For a given dataset $mathcal{D}$ and structured label $f$, the goal of Filtered Approximate Nearest Neighbor Search (FANNS) algorithms is to find top-$k$ points closest to a query that satisfy label constraints, while ensuring both recall and QPS (Queries Per Second). In recent years, many FANNS algorithms have been proposed. However, the lack of a systematic investigation makes it difficult to understand their relative strengths and weaknesses. Additionally, we found that: (1) FANNS algorithms have coupled, dataset-dependent parameters, leading to biased comparisons. (2) Key impact factors are rarely analyzed systematically, leaving unclear when each algorithm performs well. (3) Disparate datasets, workloads, and biased experiment designs make cross-algorithm comparisons unreliable. Thus, a comprehensive survey and benchmark for FANNS is crucial to achieve the following goals: designing a fair evaluation and clarifying the classification of algorithms, conducting in-depth analysis of their performance, and establishing a unified benchmark. First, we propose a taxonomy (dividing methods into extit{filter-then-search}, extit{search-then-filter}, extit{hybrid-search}) and a systematic evaluation framework, integrating unified parameter tuning and standardized filtering across algorithms to reduce implementation-induced performance variations and reflect core trade-offs. Then, we conduct a comprehensive empirical study to analyze how query difficulty and dataset properties impact performance, evaluating robustness under pressures like filter selectivity, Recall@k, and scalability to clarify each method's strengths. Finally, we establish a standardized benchmark with real-world datasets and open-source related resources to ensure reproducible future research.

Problem

• Benchmarking filtered approximate nearest neighbor search algorithms
• Analyzing performance impact factors and algorithm robustness
• Establishing standardized evaluation framework for fair comparisons

Innovation

• Proposed taxonomy categorizing FANNS algorithms
• Systematic evaluation framework with unified parameters
• Standardized benchmark with real-world datasets