Elastic Index Select for Label-Hybrid Search in Vector Database

2025-05-06

AI Summary

Existing materialized full-label-combination indexes for labeled vector nearest neighbor search suffer from exponential space and time overhead due to combinatorial explosion of label sets. Method: This paper proposes a meta-indexing framework comprising three key components: (1) modeling inclusion relationships among label sets to construct a shareable partial-order index structure; (2) introducing elastic factor theory to establish the first provable lower bound on search performance; and (3) designing a greedy index selection algorithm to optimize index composition under given space constraints. The framework is fully decoupled from underlying ANN indexes (e.g., HNSW, IVF). Contribution/Results: Experiments on multiple real-world datasets show that our approach achieves 10–500× higher search efficiency than state-of-the-art methods, approaches theoretical optimality, reduces index space exponentially, and enables cross-query index reuse.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Constraint Satisfaction and Optimization: Search

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Real-world vector embeddings are usually associated with extra labels, such as attributes and keywords. Many applications require the nearest neighbor search that contains specific labels, such as searching for product image embeddings restricted to a particular brand. A straightforward approach is to materialize all possible indices according to the complete query label workload. However, this leads to an exponential increase in both index space and processing time, which significantly limits scalability and efficiency. In this paper, we leverage the inclusion relationships among query label sets to construct partial indexes, enabling index sharing across queries for improved construction efficiency. We introduce extit{elastic factor} bounds to guarantee search performance and use the greedy algorithm to select indices that meet the bounds, achieving a tradeoff between efficiency and space. Meanwhile, we also designed the algorithm to achieve the best elastic factor under a given space limitation. Experimental results on multiple real datasets demonstrate that our algorithm can achieve near-optimal search performance, achieving up to 10x-500x search efficiency speed up over state-of-the-art approaches. Our algorithm is highly versatile, since it is not constrained by index type and can seamlessly integrate with existing optimized libraries.

Problem

• Efficiently search vector embeddings with specific labels
• Reduce exponential index space and processing time
• Balance search performance and space efficiency

Innovation

• Construct partial indexes using label set relationships
• Introduce elastic factor bounds for performance
• Greedy algorithm selects indices for efficiency-space tradeoff