🤖 AI Summary
The weak analogy between Shannon entropy and signed measures, coupled with the lack of geometric characterization for information sets, hinders a structural understanding of information. Method: We propose the Logarithmic Decomposition (LD) framework, which represents the information structure of random variables as definable “logarithmic atoms” over the sample space Ω. By extending Yeung’s I-measure, integrating signed measure theory with information geometry, and introducing the notion of logarithmic decomposability, the framework enables structured criteria for positive and negative entropy atoms. Contribution/Results: LD geometrically reconstructs common information and sufficient statistics; strictly distinguishes dyadic from triadic systems—beyond the capability of I-measure; unifies set-theoretic interpretations of mutual information, conditional entropy, and related quantities; establishes a foundation for quality-oriented information theory; and admits a natural extension to continuous distributions.
📝 Abstract
The Shannon entropy of a random variable has much behaviour analogous to a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, which are taken to represent the information that different random variables contain. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information (the intersection of sets), the joint entropy (the union of sets), and the conditional entropy (the difference of sets). Here we provide concrete characterisations of these abstract sets and a corresponding signed measure by extending the approach used by Yeung to all possible outcomes in an outcome space $Omega$, and in doing so we demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, depending only on their structure, while also being consistent with Yeung's I-measure. We present the usability of our approach by re-examining the G'acs-K""orner common information and minimally sufficient statistics from this new geometric perspective and characterising it in terms of our logarithmic atoms -- a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions before discussing implications for quality-led information theory. As a motivating example, we apply our new decomposition to the Dyadic and Triadic systems of James and Crutchfield and show that, in contrast to the I-measure alone, our decomposition is able to qualitatively distinguish between them.