Nearly Optimal Internal Dictionary Matching

2023-12-19

AI Summary

This paper studies the Internal Dictionary Matching (IDM) problem: given a text $T$ and a dictionary $mathcal{D}$ consisting of its substrings, support efficient queries on arbitrary substrings of $T$ for counting occurrences (Count), counting distinct pattern occurrences (CountDistinct), reporting distinct patterns (ReportDistinct), and deciding existence or reporting all occurrences (Exists/Report). We propose BASS, the first $O(n)$-space data structure achieving amortized $ ilde{O}(1)$ query time for CountDistinct (with $ ilde{O}(n+d)$ preprocessing), improving Count to $O(log n / log log n)$, attaining the optimal $O(1 + | ext{output}|)$ for ReportDistinct, and matching the theoretical lower bounds for Exists/Report. BASS integrates the suffix array, interval trees, bitmap compression, and blocking techniques—yielding both space efficiency and versatility—and is extensible to other internal string querying tasks.

Abstract

We study the internal dictionary matching (IDM) problem where a dictionary $mathcal{D}$ containing $d$ substrings of a text $T$ is given, and each query concerns the occurrences of patterns in $mathcal{D}$ in another substring of $T$. We propose a novel $O(n)$-sized data structure named Basic Substring Structure (BASS) where $n$ is the length of the text $T.$ With BASS, we are able to handle all types of queries in the IDM problem in nearly optimal query and preprocessing time. Specifically, our results include: $ullet$ The first algorithm that answers the CountDistinct query in $ ilde{O}(1)$ time with $ ilde{O}(n+d)$ preprocessing, where we need to compute the number of distinct patterns that exist in $T[l,r]$. Previously, the best result was $ ilde{O}(m)$ time per query after $ ilde{O}(n^2/m+d)$ or $ ilde{O}(nd/m+d)$ preprocessing, where $m$ is a chosen parameter. $ullet$ Faster algorithms for two other types of internal queries. We improve the runtime for (1) Occurrence counting (Count) queries to $O(log n/loglog n)$ time per query with $O(n+dsqrt{log n})$ preprocessing from $O(log^2 n/loglog n)$ time per query with $O(nlog n/log log n+dlog^{3/2} n)$ preprocessing. (2) Distinct pattern reporting (ReportDistinct) queries to $O(1+| ext{output}|)$ time per query from $O(log n+| ext{output}|)$ per query. In addition, we match the optimal runtime in the remaining two types of queries, pattern existence (Exists), and occurrence reporting (Report). We also show that BASS is more generally applicable to other internal query problems.

Problem

• Efficiently solve internal dictionary matching queries
• Propose BASS structure for nearly optimal query time
• Improve Count and ReportDistinct query performance

Innovation

• BASS data structure enables nearly optimal queries
• Handles CountDistinct queries in tilde{O}(1) time
• Improves Occurrence counting and Distinct pattern reporting