🤖 AI Summary
This work proposes a high-density register structure for data stream cardinality estimation that leverages arithmetic coding, relative NLZ (number of leading zeros) storage, and a shared offset mechanism to fully utilize 64-bit words and eliminate low-frequency states. Integrated with a non-iterative, four-component hybrid estimator (HLDLC) and an early-exit strategy, the approach achieves significantly improved accuracy and speed while maintaining O(B + log log C) memory scalability. Experimental results demonstrate that, under a 1 KB memory budget, the method attains a weighted mean absolute error of 1.63%, outperforming ExaLogLog by 4.7%; it also reduces the memory-variance product to 3.4 (surpassing ExaLogLog’s 3.78) and accelerates processing by 2.7–4.5×, marking the first technique to comprehensively dominate ExaLogLog across all tested memory configurations.
📝 Abstract
Cardinality estimation - counting the number of distinct elements in a data stream - requires a tradeoff between memory and accuracy. ExaLogLog recently established the state of the art for this tradeoff by combining wide registers with a Fisher-information-optimal maximum likelihood (ML) estimator, achieving the best known memory-variance product (MVP) among HyperLogLog variants. Here we present Arithmetic Variable LogLog (AVLL), which surpasses ExaLogLog at every memory point tested using arithmetic encoding and eliminating uncommon states to consume 64-bit words completely with 11 registers each, yielding a 5.5x register-count advantage. Its four-component blended estimator, HLDLC, exploits this density advantage to surpass ExaLogLog's ML accuracy without iterative solving. At 1 KB, AVLL achieves 1.63% width-weighted mean absolute error compared to ExaLogLog's 1.71% - a 4.7% improvement. The corresponding empirical MVP is 3.4, surpassing ExaLogLog's practical MVP of 3.78 and its theoretical optimum of 3.67. This holds at every tested size from 0.25 to 4 KB. AVLL inherits DynamicLogLog's early exit mechanism, which filters most elements before any register is touched. With thousands of simultaneous sketches per thread, AVLL is 2.7-4.5x faster than ExaLogLog due to the reduced memory bandwidth from early exits. Like DynamicLogLog, AVLL stores relative NLZ values with a shared offset, so its memory scales as O(B + log log C) rather than O(B x log log C) - decoupling maximum representable cardinality from register width. These results hold under both high-complexity (all-unique) and low-complexity (nonuniformly high duplication rate) data distributions, with zero accuracy degradation from duplication. AVLL is implemented as a single self-contained Java class with all correction formulas embedded, available in the BBTools suite at https://bbmap.org.