Optimal streaming algorithm for detecting $ell_2$ heavy hitters in random order streams

2025-09-08

AI Summary

This paper studies the optimal-space detection of ℓ₂-heavy hitters—items satisfying (f_i geq varepsilon sqrt{F_2})—in random-order streams. For insertion-only streams, prior algorithms fail to match the information-theoretic lower bound when (varepsilon = o(1)). We present the first algorithm achieving space complexity (Oig(frac{1}{varepsilon} log nig)) for (varepsilon = Omega(2^{-sqrt{log n}})) under both fully random-order streams (where elements arrive in uniformly random permutation) and a weaker partial random-order assumption—matching the information-theoretic lower bound. Our method integrates ideas from CountSketch and BPTree, leveraging frequency distribution properties inherent to random order: via carefully designed hash mappings and lightweight frequency estimation, it enables efficient tracking of heavy hitters with minimal overhead. Compared to the previous best algorithm, our approach saves a (log(1/varepsilon)) factor in space and achieves, for the first time, the optimal space bound for ℓ₂-heavy hitter detection in random-order streams.

Technology Category

• Search and Optimization: Heuristic Search
• Data Mining & Knowledge Management: Data Stream Mining
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Given a stream $x_1,x_2,dots,x_n$ of items from a Universe $U$ of size $mathsf{poly}(n)$, and a parameter $ε>0$, an item $iin U$ is said to be an $ell_2$ heavy hitter if its frequency $f_i$ in the stream is at least $sqrt{εF_2}$, where $F_2=sqrt{sum_{iin U} f_i^2}$. The classical $mathsf{CountSketch}$ algorithm due to Charikar, Chen, and Farach-Colton [2004], was the first algorithm to detect $ell_2$ heavy hitters using $Oleft(frac{log^2 n}ε ight)$ bits of space, and their algorithm is optimal for streams with deletions. For insertion-only streams, Braverman, Chestnut, Ivkin, Nelson, Wang, and Woodruff [2017] gave the $mathsf{BPTree}$ algorithm which requires only $Oleft(frac{log(1/ε)}εlog n ight)$ space. Note that any algorithm requires at least $Oleft(frac{1}ε log n ight)$ space to output $O(1/ε)$ heavy hitters in the worst case. So for constant $ε$, the space usage of the $mathsf{BPTree}$ algorithm is optimal but their bound could be sub-optimal for $ε=o(1)$. In this work, we show that for random order streams, where the stream elements can be adversarial but their order of arrival is uniformly random, it is possible to achieve the optimal space bound of $Oleft(frac{1}ε log n ight)$ for every $ε= Ωleft(frac{1}{2^{sqrt{log n}}} ight)$. We also show that for partially random order streams where only the heavy hitters are required to be uniformly distributed in the stream, it is possible to achieve the same space bound, but with an additional assumption that the algorithm is given a constant approximation to $F_2$ in advance.

Problem

• Detecting l2 heavy hitters in random order streams efficiently
• Achieving optimal space complexity for epsilon parameter ranges
• Improving upon existing algorithms for insertion-only data streams

Innovation

• Optimal space usage for random order streams
• Handles adversarial elements with random arrival
• Achieves O(1/ε log n) space for l2 heavy hitters