Revisiting Filtered ANN Benchmarks: A Hardness-Controlled Benchmark Generator for Realistic Evaluation

2026-06-12

AI Summary

This work addresses the lack of realism and reproducibility in existing hybrid query benchmarks—combining vector similarity search with structured filtering—which hinders robust evaluation across varying query difficulties. The authors propose α-Hardness, an execution-driven query difficulty metric that, for the first time, models the overfetching factor along the query execution pipeline as a unified, stable, and strategy-agnostic measure of hardness. Building upon this, they introduce HCBGen, a hardness-controllable benchmark generator that synthesizes workloads matching target difficulty distributions through execution-path modeling, conditional overfetch analysis, and hardness distribution fitting. Experiments reveal that prevailing benchmarks predominantly cover easy queries, whereas HCBGen effectively uncovers performance disparities among systems under high-hardness conditions and enables highly consistent proxy evaluations under privacy constraints.

Abstract

Filtered approximate nearest neighbor (FANN) search must satisfy both vector similarity and structured predicates, yet evaluations remain brittle because real hybrid workloads are rarely shareable and existing benchmarks rely on ad-hoc synthetic or semi-real constructions. We argue that realism hinges on execution-driven query difficulty: failures in early filtering trigger over-fetching of additional candidates, shaping latency, throughput, and recall. Building on this insight, we propose $α$-Hardness, a query-level hardness metric that models the conditional execution chain via the over-fetch factor and extends naturally to strategy-conditioned settings. Across diverse datasets and hybrid strategies, $α$-Hardness exhibits strong monotonic alignment with empirical performance, while common proxies such as selectivity or attribute-vector correlation are frequently unstable or strategy-inconsistent. We further introduce HCBGen, a hardness-controlled benchmark generator that uses $α$-Hardness as an explicit control signal to synthesize workloads under coarse bias modes or to match a target hardness profile. Our experiments show that widely used benchmarks occupy a narrow, relatively easy portion of the hardness spectrum, masking robustness gaps that emerge under harder queries. Finally, we demonstrate that matching hardness distributions enables privacy-preserving proxy workloads that closely reproduce performance trends, bridging research benchmarks and real evaluation.

Problem

• filtered approximate nearest neighbor
• benchmark realism
• query hardness
• hybrid workloads
• performance evaluation

Innovation

• Filtered ANN
• query hardness
• benchmark generation
• over-fetch factor
• hybrid search