Evaluating and Generating Query Workloads for High Dimensional Vector Similarity Search

2026-06-12

AI Summary

Existing benchmarks for vector similarity search lack effective control over query difficulty, undermining the accuracy of algorithmic evaluation. This work presents the first systematic assessment of the correlation between various query difficulty metrics and actual retrieval costs, and introduces two novel methods for generating query workloads with specified difficulty levels: HephAnn, based on anchor perturbation, and HephGrad, driven by optimization. Experimental results demonstrate that HephGrad achieves higher efficiency, while HephAnn offers broader applicability; both methods reliably produce workloads matching target difficulty levels. Consequently, they significantly enhance the controllability, reproducibility, and fine-grained reliability of benchmarking evaluations in vector similarity search.

Abstract

Similarity search lies at the heart of many modern applications, ranging from databases to deep learning to data series analysis. As such, a vast effort has been invested in developing algorithms, data structures and implementations to speed up this crucial subroutine. To empirically validate these approaches, several benchmarking efforts have been initiated covering a wide array of datasets. In this paper, we observe that usually little control is exercised on the hardness of the workloads with which methods are tested and compared. To address this issue, we first evaluate several query hardness measures with respect to their ability to capture the empirical hardness of a query, i.e. the effort invested by an index data structure to provide an answer. Then, we propose two methods, deemed \HephAnn and \HephGrad, for synthesizing query workloads so that they meet a user-specified hardness target. Both methods allow to produce workloads with the desired hardness: we find that \HephGrad is faster, while \HephAnn makes fewer assumptions on the target hardness measure. The resulting workloads can be used to gain insights into the behavior of similarity search algorithms.

Problem

• vector similarity search
• query workload
• hardness evaluation
• high-dimensional data
• benchmarking

Innovation

• query workload synthesis
• vector similarity search
• query hardness
• HephAnn
• HephGrad