Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search

2025-01-16

AI Summary

In approximate nearest neighbor (ANN) search, metadata—such as vector identifiers and graph edges—incurs substantial memory overhead, hindering scalable deployment of large-scale indexes. Method: This paper proposes a lossless, efficient, zero-precision-loss vector ID compression scheme. It pioneers the integration of Asymmetric Numeral Systems (ANS) coding with wavelet tree structures for ID compression, and jointly models quantization error using the structural properties of inverted files and graph indexes; leveraging inherent suboptimality in quantization algorithms to achieve additional lossless codebook compression. Results: Experiments show a 7× compression ratio on vector IDs, a 30% reduction in overall index size for billion-scale datasets, zero accuracy degradation in query results, and no increase in query latency. The method significantly improves storage efficiency and scalability of ANN indexes without compromising speed or accuracy.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine. Lossy vector compression, i.e., embedding quantization, has been applied extensively to reduce the size of indexes. However, for inverted file and graph-based indices, auxiliary data such as vector ids and links (edges) can represent most of the storage cost. We introduce and evaluate lossless compression schemes for these cases. These approaches are based on asymmetric numeral systems or wavelet trees that exploit the fact that the ordering of ids is irrelevant within the data structures. In some settings, we are able to compress the vector ids by a factor 7, with no impact on accuracy or search runtime. On billion-scale datasets, this results in a reduction of 30% of the index size. Furthermore, we show that for some datasets, these methods can also compress the quantized vector codes losslessly, by exploiting sub-optimalities in the original quantization algorithm. The source code for our approach available at https://github.com/facebookresearch/vector_db_id_compression.

Problem

• Large-scale Database
• Vector ID Compression
• Similarity Search

Innovation

• Vector Data Compression
• Index Size Reduction
• Secondary Compression