A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective

2026-02-20

AI Summary

This study addresses the challenge of preserving predictive information about a target variable while removing irrelevant redundancy in data compression. Building on statistical decision theory, the authors propose an ℋ-mutual information framework that satisfies conditional independence (CV) and average generalization (AVG) criteria. They establish, for the first time, an equivalence between the generalized information bottleneck problem and Expected Sample Information (ESI), thereby enabling a computable characterization of a representation’s predictive utility. An alternating optimization algorithm is further developed to efficiently approximate the Pareto frontier between compression and utility. This work extends the applicability of classical mutual information and offers a new paradigm for information bottleneck theory that balances theoretical rigor with practical utility.

Technology Category

• Machine Learning: Information Theory
• Data Mining & Knowledge Management: Data Compression
• Reasoning under Uncertainty: Decision/Utility Theory

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Content-based information diffusion
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

The information bottleneck (IB) method seeks a compressed representation of data that preserves information relevant to a target variable for prediction while discarding irrelevant information from the original data. In its classical formulation, the IB method employs mutual information to evaluate the compression between the original and compressed data and the utility of the representation for the target variable. In this study, we investigate a generalized IB problem, where the evaluation of utility is based on the $\mathcal{H}$-mutual information that satisfies the concave (\texttt{CV}) and averaging (\texttt{AVG}) conditions. This class of information measures admits a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information. Based on this interpretation, we derive an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem.

Problem

• information bottleneck
• data compression
• utility
• H-mutual information
• decision theory

Innovation

• Generalized Information Bottleneck
• H-mutual information
• Decision-theoretic interpretation
• Expected Value of Sample Information
• Alternating optimization