Multimodal Concept Bottleneck Models

2026-06-18

AI Summary

This work addresses the limited generalization and unintended leakage of non-conceptual information in traditional concept bottleneck models, which hinder their applicability in open-world settings. The authors propose the first integration of concept bottleneck mechanisms into a multimodal framework such as CLIP, introducing a dual-modality concept bottleneck layer that jointly aligns image and text embeddings into an interpretable concept space. By combining concept-level supervision with end-to-end training, the method effectively suppresses non-conceptual signal leakage while enabling open-vocabulary tasks like zero-shot classification and image retrieval. Evaluated across four benchmarks, the approach achieves an average accuracy of 95% relative to black-box models—an absolute improvement of 51.26%—demonstrating a significant balance between predictive performance and model interpretability.

Abstract

Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts. However, existing CBMs are constrained in their ability to generalize beyond a fixed set of predefined classes and the risk of non-concept information leakage, where predictive signals outside the intended concepts are inadvertently exploited. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks. Our method maintains high accuracy, staying within ~5% of black-box performance while offering greater interpretability.

Problem

• Concept Bottleneck Models
• interpretability
• generalization
• information leakage
• multimodal learning

Innovation

• Multimodal Concept Bottleneck Model
• Concept Bottleneck Layer
• Interpretable AI
• CLIP
• Zero-shot Classification