Theory Meets Practice for Bit Vectors Supporting Rank and Select

2025-09-22

AI Summary

This work addresses the gap between theoretical optimality and practical efficiency for rank/select operations on bit vectors. We propose a compact data structure achieving worst-case O(1) query time with only 0.78% space overhead. Methodologically, it is the first to provide rigorous theoretical guarantees of constant-time rank/select via a hierarchical indexing scheme coupled with bit-level storage optimizations—enabling hardware-friendly bit manipulation while maintaining minimal redundancy. Compared to state-of-the-art alternatives, our design reduces space overhead by 4.5×–64.1× and delivers significantly faster query performance on real-world datasets. By bridging the gap between asymptotic lower bounds and engineering constraints, this work establishes the currently best practical solution for large-scale bitmap indexing, compressed data structures, and related applications in databases and information retrieval.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Distributed Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Bit vectors with support for fast rank and select are a fundamental building block for compressed data structures. We close a gap between theory and practice by analyzing an important part of the design space and experimentally evaluating a sweet spot. The result is the first implementation of a rank and select data structure for bit vectors with worst-case constant query time, good practical performance, and a space-overhead of just 0.78%, i.e., between $4.5 imes$ and $64.1 imes$ less than previous implementations.

Problem

• Bridging theory-practice gap for rank/select bit vectors
• Achieving worst-case constant query time with minimal space
• Reducing space overhead significantly versus prior implementations

Innovation

• Worst-case constant query time implementation
• Practical performance with 0.78% space overhead
• Experimental evaluation of design space sweet spot