Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures

2025-08-07

AI Summary

Multivariate partial information decomposition (PID) suffers from fundamental axiom inconsistencies in settings with three or more information sources. Method: We first derive a closed-form solution for the two-source PID that satisfies all canonical axioms; then prove that multivariate PID admits intrinsic, unsolvable contradictions under the standard lattice structure, exposing its theoretical limitations. To overcome this, we propose a novel, lattice-free framework: by reconstructing random variables to eliminate higher-order dependencies, we define unique and synergistic information measures that are both additive and continuous. Contribution/Results: We rigorously prove that these measures satisfy key PID axioms—including non-negativity, identity, and monotonicity—while avoiding lattice-induced inconsistencies. Systematic experiments on Ising models demonstrate their numerical stability and interpretability. This work provides the first quantification framework for higher-order information interactions that is both theoretically sound and practically reliable.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

While mutual information effectively quantifies dependence between two variables, it cannot capture complex, fine-grained interactions that emerge in multivariate systems.The Partial Information Decomposition (PID) framework was introduced to address this by decomposing the mutual information between a set of source variables and a target variable into fine-grained information atoms such as redundant, unique, and synergistic components. In this work, we review the axiomatic system and desired properties of the PID framework and make three main contributions. First, we resolve the two-source PID case by providing explicit closed-form formulas for all information atoms that satisfy the full set of axioms and desirable properties. Second, we prove that for three or more sources, PID suffers from fundamental inconsistencies: we present a three-variable counterexample where the sum of atoms exceeds the total information, and prove an impossibility theorem showing that no lattice-based decomposition can be consistent for all subsets when the number of sources exceeds three. Finally, we deviate from the PID lattice approach to avoid its inconsistencies, and present explicit measures of multivariate unique and synergistic information. Our proposed measures, which rely on new systems of random variables that eliminate higher-order dependencies, satisfy key axioms such as additivity and continuity, provide a robust theoretical explanation of high-order relations, and show strong numerical performance in comprehensive experiments on the Ising model. Our findings highlight the need for a new framework for studying multivariate information decomposition.

Problem

• Resolve inconsistencies in two-source Partial Information Decomposition
• Prove impossibility of consistent lattice-based decomposition for three+ sources
• Propose new measures for multivariate unique and synergistic information

Innovation

• Closed-form formulas for two-source PID
• Impossibility theorem for three-source PID
• New measures for multivariate information decomposition