The Trie Measure, Revisited

2025-04-14

AI Summary

This paper addresses the prefix-free encoding optimization problem for families of integer sets: given $n$ subsets $S_1,dots,S_n$ of a universal set $U$ (with total cardinality $N$), find an encoding $ ext{enc}$ that minimizes the sum of edge counts across the binary tries induced by the encoded sets. We propose two novel algorithms: (1) an optimal shift-invariant fixed-length encoding, running in $O(u + N log u)$ time, where $u = |U|$; and (2) the first $O(N + u^3)$-time algorithm for optimal *ordered* encoding—proven to dominate both separately optimized fixed-length and ordered encodings. Our approach integrates modular arithmetic, dynamic programming, and trie structural analysis. Experimental results demonstrate that our encodings significantly reduce trie size, establishing an efficient encoding foundation for large-scale integer set indexing.

Technology Category

• Search and Optimization: Combinatorial Optimization
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In this paper, we study the following problem: given $n$ subsets $S_1, dots, S_n$ of an integer universe $U = {0,dots, u-1}$, having total cardinality $N = sum_{i=1}^n |S_i|$, find a prefix-free encoding $enc : U ightarrow {0,1}^+$ minimizing the so-called trie measure, i.e., the total number of edges in the $n$ binary tries $mathcal T_1, dots, mathcal T_n$, where $mathcal T_i$ is the trie packing the encoded integers ${enc(x):xin S_i}$. We first observe that this problem is equivalent to that of merging $u$ sets with the cheapest sequence of binary unions, a problem which in [Ghosh et al., ICDCS 2015] is shown to be NP-hard. Motivated by the hardness of the general problem, we focus on particular families of prefix-free encodings. We start by studying the fixed-length shifted encoding of [Gupta et al., Theoretical Computer Science 2007]. Given a parameter $0le a<u$, this encoding sends each $x in U$ to $(x + a) mod u$, interpreted as a bit-string of $log u$ bits. We develop the first efficient algorithms that find the value of $a$ minimizing the trie measure when this encoding is used. Our two algorithms run in $O(u + Nlog u)$ and $O(Nlog^2 u)$ time, respectively. We proceed by studying ordered encodings (a.k.a. monotone or alphabetic), and describe an algorithm finding the optimal such encoding in $O(N+u^3)$ time. Within the same running time, we show how to compute the best shifted ordered encoding, provably no worse than both the optimal shifted and optimal ordered encodings. We provide implementations of our algorithms and discuss how these encodings perform in practice.

Problem

• Find prefix-free encoding minimizing trie edges for subsets
• Develop efficient algorithms for optimal shifted encoding
• Compute best ordered and shifted ordered encodings efficiently

Innovation

• Efficient algorithms for shifted encoding optimization
• Optimal ordered encoding in O(N+u^3) time
• Combined shifted and ordered encoding optimization