Succinct Dynamic Rank/Select: Bypassing the Tree-Structure Bottleneck

2025-10-21

AI Summary

This paper addresses the space–time optimality of dynamic ordered dictionaries: maintaining $n$ elements from a universe of size $U = mathrm{poly}(n)$, supporting insertions, deletions, rank, and select operations. We introduce the first construction to break the “tree-structure bottleneck”—the compressed lookup-table weighted Treap—which achieves $o(n)$-bit redundancy (specifically $O(log U)$, or $O(1)$ when $n$ and $U$ are known) in the dynamic setting, attaining near-entropy-optimal space $log inom{U}{n} + o(n)$. Time complexity is optimal amortized $Oig(1 + log n / log log Uig)$. Our result positively resolves an open problem posed by Pibiri and Venturini, establishing a new theoretical benchmark for dynamic order-statistic data structures.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Data Compression
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

We show how to construct a dynamic ordered dictionary, supporting insert/delete/rank/select on a set of $n$ elements from a universe of size $U$, that achieves the optimal amortized expected time complexity of $O(1 + log n / log log U)$, while achieving a nearly optimal space consumption of $log inom{U}{n} + n / 2^{(log n)^{Ω(1)}} + ext{polylog}, U$ bits in the regime where $U = ext{poly}(n)$. This resolves an open question by Pibiri and Venturini as to whether a redundancy (a.k.a. space overhead) of $o(n)$ bits is possible, and is the first dynamic solution to bypass the so-called tree-structure bottleneck, in which the bits needed to encode some dynamic tree structure are themselves enough to force a redundancy of $widetildeΩ(n)$ bits. Our main technical building block is a dynamic balanced binary search tree, which we call the compressed tabulation-weighted treap, that itself achieves a surprising time/space tradeoff. The tree supports $ ext{polylog}, n$-time operations and requires a static lookup table of size $ ext{poly}(n) + ext{polylog}, U$ -- but, in exchange for these, the tree is able to achieve a remarkable space guarantee. Its total space redundancy is $O(log U)$ bits. In fact, if the tree is given $n$ and $U$ for free, then the redundancy further drops to $O(1)$ bits.

Problem

• Bypassing the tree-structure bottleneck in dynamic ordered dictionaries
• Achieving optimal time complexity with nearly minimal space overhead
• Resolving whether o(n) bits redundancy is possible for dynamic rank/select

Innovation

• Dynamic balanced binary search tree with compressed tabulation-weighted treap
• Achieves optimal time complexity with minimal space overhead
• Bypasses tree-structure bottleneck using static lookup table