Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

2022-02-18

arXiv.org

AI Summary

Classical information diagram theory has limited applicability due to its reliance on Shannon entropy and restrictive assumptions. Method: This paper proposes a generalized information decomposition framework axiomatized via monoid actions and the chain rule, unifying Hu’s theorem across diverse information measures—including Tsallis entropy, KL divergence, Kolmogorov complexity, and machine learning generalization error—for the first time. The approach integrates abstract algebra, probabilistic asymptotic analysis, and submodular function theory. Contributions/Results: (1) It establishes asymptotic equivalence between algorithmic information theory and classical information theory; (2) it proves that the expected interaction complexity converges to Shannon interaction information; and (3) it reveals asymptotic consistency between per-bit expected interaction complexity and information content for long sequences. By transcending Shannon-centric foundations, this framework substantially extends the theoretical scope and practical applicability of information diagrams in modeling complex systems.

Abstract

In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual information, and higher interaction information relate to each other. This is answered by Hu's theorem, which is widely known in the form of information diagrams: it relates shapes in a Venn diagram to information functions, thus establishing a bridge from set theory to information theory. In this work, we view random variables together with the joint operation as a monoid that acts by conditioning on information functions, and entropy as a function satisfying the chain rule of information. This abstract viewpoint allows to prove a generalization of Hu's theorem. It applies to Shannon and Tsallis entropy, (Tsallis) Kullback-Leibler Divergence, cross-entropy, Kolmogorov complexity, submodular information functions, and the generalization error in machine learning. Our result implies for Chaitin's Kolmogorov complexity that the interaction complexities of all degrees are in expectation close to Shannon interaction information. For well-behaved probability distributions on increasing sequence lengths, this shows that the per-bit expected interaction complexity and information asymptotically coincide, thus showing a strong bridge between algorithmic and classical information theory.

Problem

• Information Theory
• Entropy Measures
• Complexity Theory

Innovation

• Unified Information Measures
• Extended Hu's Theorem
• Algorithmic Information Theory Convergence