Indexing Strings with Utilities

2025-04-08

AI Summary

This paper introduces the Practical String Indexing (USI) problem: preprocessing long texts—each character associated with a positional utility value (e.g., numerical scores in biological sequences or ad logs)—to efficiently support global utility aggregation queries under pattern matching. It is the first work to incorporate the notion of “utility” into string indexing. The authors propose a compact trie-based index structure built upon top-K frequent substrings and design two novel algorithms: (i) a linear-space substring mining algorithm, and (ii) a space-sensitive utility estimation algorithm—both specifically tailored to overcome the limitations of conventional itemset mining methods in substring contexts. Experiments on billion-character datasets demonstrate that the substring mining is accurate and scalable, while USI achieves up to 15× speedup over baseline approaches and significantly outperforms them under comparable space budgets.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Distributed Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Topic discovery and tracking
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Applications in domains ranging from bioinformatics to advertising feature strings that come with numerical scores (utilities). The utilities quantify the importance, interest, profit, or risk of the letters occurring at every position of a string. Motivated by the ever-increasing rate of generating such data, as well as by their importance in several domains, we introduce Useful String Indexing (USI), a natural generalization of the classic String Indexing problem. Given a string $S$ (the text) of length $n$, USI asks for preprocessing $S$ into a compact data structure supporting the following queries efficiently: given a shorter string $P$ (the pattern), return the global utility $U(P)$ of $P$ in $S$, where $U$ is a function that maps any string $P$ to a utility score based on the utilities of the letters of every occurrence of $P$ in $S$. Our work also makes the following contributions: (1) We propose a novel and efficient data structure for USI based on finding the top-$K$ frequent substrings of $S$. (2) We propose a linear-space data structure that can be used to mine the top-$K$ frequent substrings of $S$ or to tune the parameters of the USI data structure. (3) We propose a novel space-efficient algorithm for estimating the set of the top-$K$ frequent substrings of $S$, thus improving the construction space of the data structure for USI. (4) We show that popular space-efficient top-$K$ frequent item mining strategies employed by state-of-the-art algorithms do not smoothly translate from items to substrings. (5) Using billion-letter datasets, we experimentally demonstrate that: (i) our top-$K$ frequent substring mining algorithms are accurate and scalable, unlike two state-of-the-art methods; and (ii) our USI data structures are up to $15$ times faster in querying than $4$ nontrivial baselines while occupying the same space with them.

Problem

• Indexing strings with numerical utilities for efficient querying
• Finding top-K frequent substrings to optimize data structure
• Improving space efficiency in top-K substring mining algorithms

Innovation

• Introduces Useful String Indexing (USI) for scored strings
• Proposes efficient top-K frequent substring mining
• Develops space-efficient USI query structures