Is the Information Bottleneck Robust Enough? Towards Label-Noise Resistant Information Bottleneck Learning

2025-12-11

AI Summary

The Information Bottleneck (IB) principle suffers from severe overfitting and performance degradation under label noise due to its reliance on exact ground-truth labels. To address this, we propose LaT-IB, a label-noise-robust IB learning framework grounded in the novel “Minimal–Sufficient–Clean” (MSC) principle, which theoretically guarantees separation of clean label information from noise components. LaT-IB introduces a noise-aware latent disentanglement mechanism and a three-stage progressive training strategy—Warmup, Knowledge Injection, and Robust Optimization—integrated with mutual information regularization and disentangled representation learning. Extensive experiments across diverse noise settings demonstrate that LaT-IB consistently outperforms existing IB-based and robust learning methods, achieving significant improvements in classification accuracy and generalization stability. These results validate its effectiveness and practicality in real-world noisy-label scenarios.

Technology Category

• Machine Learning: Information Theory
• Intelligent Robots: Learning & Optimization for ROB
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information. However, its strong reliance on accurate labels makes it inherently vulnerable to label noise, prevalent in real-world scenarios, resulting in significant performance degradation and overfitting. To address this issue, we propose LaT-IB, a novel Label-Noise ResistanT Information Bottleneck method which introduces a "Minimal-Sufficient-Clean" (MSC) criterion. Instantiated as a mutual information regularizer to retain task-relevant information while discarding noise, MSC addresses standard IB's vulnerability to noisy label supervision. To achieve this, LaT-IB employs a noise-aware latent disentanglement that decomposes the latent representation into components aligned with to the clean label space and the noise space. Theoretically, we first derive mutual information bounds for each component of our objective including prediction, compression, and disentanglement, and moreover prove that optimizing it encourages representations invariant to input noise and separates clean and noisy label information. Furthermore, we design a three-phase training framework: Warmup, Knowledge Injection and Robust Training, to progressively guide the model toward noise-resistant representations. Extensive experiments demonstrate that LaT-IB achieves superior robustness and efficiency under label noise, significantly enhancing robustness and applicability in real-world scenarios with label noise.

Problem

• Addresses Information Bottleneck's vulnerability to label noise
• Proposes a noise-resistant method with latent disentanglement
• Enhances robustness in real-world scenarios with noisy labels

Innovation

• Introduces Minimal-Sufficient-Clean criterion via mutual information regularizer
• Employs noise-aware latent disentanglement to separate clean and noisy information
• Uses three-phase training framework for progressive noise-resistant representation learning