Optimal Static Fully Indexable Dictionaries

2025-04-27

AI Summary

This work addresses the long-standing space–time trade-off bottleneck for static fully indexable dictionaries (FIDs) and partial-sum data structures under large universes (U = n^{1+Θ(1)}). We propose the first compact and query-optimal solution by integrating hierarchical hashing, recursive compression, and bit-level encoding—guided by information-theoretic lower bound analysis. Our FID achieves space log binom(U,n) + n/(log U/t)^{Ω(t)} bits and worst-case O(t + log log n) time for rank/select queries, thereby unifying asymptotically tight space and matching lower-bound query time for the first time. Concurrently, we break the longstanding barrier for the partial-sum problem: prior constructions required either constant query time with Ω(n) redundancy or subconstant redundancy with superconstant query time. Our structure supports O(t) query time using total space nℓ + n/(log n/t)^{Ω(t)} bits—achieving sublinear redundancy while supporting tunable, near-constant query time.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

Fully indexable dictionaries (FID) store sets of integer keys while supporting rank/select queries. They serve as basic building blocks in many succinct data structures. Despite the great importance of FIDs, no known FID is succinct with efficient query time when the universe size $U$ is a large polynomial in the number of keys $n$, which is the conventional parameter regime for dictionary problems. In this paper, we design an FID that uses $log inom{U}{n} + frac{n}{(log U / t)^{Omega(t)}}$ bits of space, and answers rank/select queries in $O(t + log log n)$ time in the worst case, for any parameter $1 le t le log n / log log n$, provided $U = n^{1 + Theta(1)}$. This time-space trade-off matches known lower bounds for FIDs [Pv{a}trac{s}cu&Thorup STOC 2006; Pv{a}trac{s}cu&Viola SODA 2010] when $t le log^{0.99} n$. Our techniques also lead to efficient succinct data structures for the fundamental problem of maintaining $n$ integers each of $ell = Theta(log n)$ bits and supporting partial-sum queries, with a trade-off between $O(t)$ query time and $nell + n / (log n / t)^{Omega(t)}$ bits of space. Prior to this work, no known data structure for the partial-sum problem achieves constant query time with $n ell + o(n)$ bits of space usage.

Problem

• Designing succinct FID for large universe size U.
• Achieving optimal time-space trade-off for rank/select queries.
• Developing efficient succinct data structures for partial-sum queries.

Innovation

• Optimal static fully indexable dictionaries design
• Efficient rank/select queries in O(t) time
• Succinct data structures for partial-sum queries