Information Subtraction: Learning Representations for Conditional Entropy

2025-01-02

AI Summary

This work addresses the challenge of modeling conditional entropy and conditional mutual information for continuous sensitive variables. We propose Info-Subtraction—the first conditional information disentanglement framework tailored to continuous variables. Methodologically, it leverages a generative information bottleneck to jointly optimize the target mutual information $I(X;Z)$ and sensitive mutual information $I(S;Z)$ in the latent space, while introducing a novel continuous conditional contrastive mechanism that enables iterative extraction of semantic features such as conditional entropy. Our key contributions are: (i) the first theoretical extension of conditional information disentanglement to continuous-variable settings, yielding a principled, interpretable foundation for fair representation learning; (ii) significant performance gains on fair classification and cross-domain recognition benchmarks; and (iii) empirically verified high-fidelity encoding of conditional entropy semantics in the learned representations. The implementation is open-sourced and has been adopted by the research community.

Technology Category

• Machine Learning: Information Theory
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Information Extraction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction

Problem

• Conditional Entropy
• Conditional Mutual Information
• Fair Learning

Innovation

• Information Subtraction
• Continuous Variables
• Fair Learning