Information-Theoretic Measures on Lattices for High-Order Interactions

2024-08-14

arXiv.org

AI Summary

Traditional pairwise information measures fail to capture higher-order statistical dependencies among multivariate variables; existing higher-order approaches suffer from computational intractability, asymmetry, or failure to account for all possible joint distribution decompositions. This paper proposes the first systematic framework for rigorously defining full-order information interaction among $d$ variables, grounded in partition lattice structure and operator function pairs. We introduce Streitberg information—a symmetric, decomposable measure defined over the complete partition lattice via a generalized KL divergence—capable of unbiasedly quantifying interactions of all orders for $d > 3$ while avoiding cancellation effects. Integrating lattice theory, information theory, and numerical optimization, we validate theoretical correctness on synthetic data and demonstrate substantial improvements in robustness and interpretability across financial dependency analysis, neural signal decoding, and high-dimensional feature selection.

Technology Category

• Machine Learning: Information Theory
• Natural Language Processing: Information Extraction
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Traditional models based solely on pairwise associations often fail to capture the complex statistical structure of multivariate data. Existing approaches for identifying information shared among groups of $d>3$ variables are frequently computationally intractable, asymmetric with respect to a target variable, or unable to account for all factorisations of the joint probability distribution. We present a systematic framework that derives higher-order information-theoretic measures using lattice and operator function pairs, whereby the lattice representing the algebraic relationships among variables, with operator functions that compute the measures over the lattice. We show that many commonly used measures can be derived within this framework, however they are often restricted to sublattices of the partition lattice, which prevents them from capturing all interactions when $d>3$. We also demonstrate that KL divergence, when used as an operator function, leads to unwanted cancellation of interactions for $d>3$. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, using generalisations of KL divergence as an operator function, and defined over the full partition lattice. We validate Streitberg Information numerically on synthetic data, and illustrate its application in analysing complex interactions among stocks, decoding neural signals, and performing feature selection in machine learning.

Problem

• Traditional measures fail to capture multivariate data complexity.
• Existing methods for d>3 variables are computationally intractable or incomplete.
• Streitberg Information introduced to fully characterize d-variable interactions.

Innovation

• Lattice theory framework for higher-order measures
• Streitberg Information captures all variable interactions
• Generalised divergences beyond KL for operator functions