Tight Lower Bounds for Central String Queries in Compressed Space

2025-10-22

AI Summary

This paper investigates time-complexity lower bounds for fundamental queries—such as suffix array (SA), longest-common-prefix (LCP), longest-common-extension (LCE), Burrows–Wheeler transform (BWT), LF-mapping, permuted LCP (PLCP), and Φ-function—on compressed string indexes. Leveraging δ(T), a tight measure of substring complexity, the authors combine information-theoretic and computational-complexity arguments to construct asymptotically tight lower-bound models over binary alphabets. They establish, for the first time, two strictly separated optimal time bounds: Ω(log n / log log n) for SA/LCP/LCE queries, and Ω(log log n) for BWT/LF/PLCP/Φ queries—yielding a clean “dichotomous boundary” theory. This resolves a long-standing gap between upper and lower bounds in compressed indexing, completes the foundational theoretical framework for compressed string indexes, and reveals the intrinsic minimal time cost of basic queries under near-optimal compression.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Combinatorial Optimization
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Applications of cryptography
• Web Mining and Content Analysis: Models for Web evolution

Abstract

In this work, we study the limits of compressed data structures, i.e., structures that support various queries on an input text $TinSigma^n$ using space proportional to the size of $T$ in compressed form. Nearly all fundamental queries can currently be efficiently supported in $O(delta(T)log^{O(1)}n)$ space, where $delta(T)$ is the substring complexity, a strong compressibility measure that lower-bounds the optimal space to represent the text [Kociumaka, Navarro, Prezza, IEEE Trans. Inf. Theory 2023]. However, optimal query time has been characterized only for random access. We address this gap by developing tight lower bounds for nearly all other fundamental queries: (1) We prove that suffix array (SA), inverse suffix array (SA$^{-1}$), longest common prefix (LCP) array, and longest common extension (LCE) queries all require $Omega(log n/loglog n)$ time within $O(delta(T)log^{O(1)}n)$ space, matching known upper bounds. (2) We further show that other common queries, currently supported in $O(loglog n)$ time and $O(delta(T)log^{O(1)}n)$ space, including the Burrows-Wheeler Transform (BWT), permuted longest common prefix (PLCP) array, Last-to-First (LF), inverse LF, lexicographic predecessor ($Phi$), and inverse $Phi$ queries, all require $Omega(loglog n)$ time, yielding another set of tight bounds. Our lower bounds hold even for texts over a binary alphabet. This work establishes a clean dichotomy: the optimal time complexity to support central string queries in compressed space is either $Theta(log n/loglog n)$ or $Theta(loglog n)$. This completes the theoretical foundation of compressed indexing, closing a crucial gap between upper and lower bounds and providing a clear target for future data structures: seeking either the optimal time in the smallest space or the fastest time in the optimal space, both of which are now known for central string queries.

Problem

• Establishing tight time lower bounds for string queries in compressed space
• Proving optimal query times for suffix array and LCP array operations
• Characterizing time complexity dichotomy for compressed string indexing

Innovation

• Proved tight lower bounds for SA and LCP queries
• Established optimal time dichotomy for compressed queries
• Closed theoretical gap between upper and lower bounds