A Family of Lz78-Based Universal Sequential Probability Assignments

2024-10-09

International Symposium on Information Theory

AI Summary

This paper addresses universal probability assignment for individual sequences. We propose a family of probability assignment methods based on incremental LZ78 dictionary parsing. Our key contribution is the first rigorous establishment of uniform convergence between LZ78 codeword length and normalized log-loss: we prove that the normalized log-loss of these methods converges uniformly to the LZ78 code length, thereby achieving asymptotically optimal compression. This result guarantees strong universality for finite-state sources and Markov sources. Unlike conventional approaches relying on parametric statistical models or Bayesian priors, our method is fully data-driven and imposes no assumptions about source structure. Empirical evaluations demonstrate its practical effectiveness in sequence generation and classification tasks. Overall, this work provides a novel paradigm for universal source coding and model-free probabilistic modeling.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

We propose and study a family of universal Sequential Probability Assignments (SPAs) on individual sequences, based on the incremental parsing procedure of the Lempel-Ziv (LZ78) compression algorithm. We show that the normalized log loss under any of these models converges to the normalized LZ78 codelength, uniformly over all individual sequences. To establish the universality of these models, we consolidate a set of results from the literature relating finite-state compressibility to optimal log-loss under Markovian and finite-state models. Finally, we present experimental results showcasing the potential of the LZ78 SPA family for generative and classification tasks.

Problem

• Develops LZ78-based universal sequential probability assignments
• Demonstrates convergence to LZ78 codelength across sequences
• Explores applications in compression, generation, and classification

Innovation

• LZ78-based universal sequential probability assignments
• Normalized log loss converges to LZ78 codelength
• Models applied to compression, generation, and classification