Separable Computation of Information Measures

2025-01-25

AI Summary

This study addresses the challenge of directly estimating information-theoretic measures—such as mutual information, f-information, and Wyner’s common information—in high-dimensional settings. Methodologically, it proposes a learnable, separable computation framework: for feature representations satisfying mild regularity conditions, these information measures decompose into independent terms estimable separately at the feature level and the raw-data level. The framework unifies representation learning, f-divergence theory, and common information modeling into a coherent analytical paradigm for information measurement. Theoretically, it establishes novel connections between information measures and underlying statistical dependence structures, providing rigorous theoretical guarantees for information bottleneck methods and related principles. Empirically, the approach significantly enhances the feasibility, robustness, and interpretability of information estimation in high-dimensional scenarios, enabling reliable quantification where conventional estimators fail.

Technology Category

• Machine Learning: Information Theory
• Natural Language Processing: Information Extraction
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Web measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. Under mild assumptions on the feature representations, we demonstrate that a class of information measures admit such separable computation, including mutual information, $f$-information, Wyner's common information, G{'a}cs--K{""o}rner common information, and Tishby's information bottleneck. Our development establishes several new connections between information measures and the statistical dependence structure. The characterizations also provide theoretical guarantees of practical designs for estimating information measures through representation learning.

Problem

• Information Quantification
• Learned Features
• Theoretical Framework

Innovation

• Information Quantification
• Learned Features
• Statistical Relationship