Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability

2025-02-16

AI Summary

To address the insufficient interpretability of multimodal image-text models (e.g., CLIP) in high-stakes domains such as healthcare, this paper proposes the first native multimodal, deterministic, and verifiable attribution framework. Departing from conventional information bottleneck assumptions involving redundant compression, we introduce the Narrowing Information Bottleneck theory, which rigorously satisfies modern attribution axioms—including completeness and sensitivity. Our method reformulates the objective function via information-theoretic principles, integrating gradient regularization with semantic consistency constraints, and employs a modular bidirectional attribution propagation mechanism to achieve high-fidelity joint explanations for both images and text. Experiments demonstrate significant improvements: +9% in image explanation accuracy, +58.83% in text explanation accuracy, and a 63.95% speedup in inference. The implementation is publicly available.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Multimodal Learning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The task of identifying multimodal image-text representations has garnered increasing attention, particularly with models such as CLIP (Contrastive Language-Image Pretraining), which demonstrate exceptional performance in learning complex associations between images and text. Despite these advancements, ensuring the interpretability of such models is paramount for their safe deployment in real-world applications, such as healthcare. While numerous interpretability methods have been developed for unimodal tasks, these approaches often fail to transfer effectively to multimodal contexts due to inherent differences in the representation structures. Bottleneck methods, well-established in information theory, have been applied to enhance CLIP's interpretability. However, they are often hindered by strong assumptions or intrinsic randomness. To overcome these challenges, we propose the Narrowing Information Bottleneck Theory, a novel framework that fundamentally redefines the traditional bottleneck approach. This theory is specifically designed to satisfy contemporary attribution axioms, providing a more robust and reliable solution for improving the interpretability of multimodal models. In our experiments, compared to state-of-the-art methods, our approach enhances image interpretability by an average of 9%, text interpretability by an average of 58.83%, and accelerates processing speed by 63.95%. Our code is publicly accessible at https://github.com/LMBTough/NIB.

Problem

• Enhancing multimodal model interpretability
• Overcoming limitations in bottleneck methods
• Improving image-text representation clarity

Innovation

• Narrowing Information Bottleneck Theory
• Enhances multimodal model interpretability
• Increases processing speed significantly