Rethinking Concept Bottleneck Models: From Pitfalls to Solutions

2026-03-05

AI Summary

This work addresses critical limitations of Concept Bottleneck Models (CBMs), including unpredictable concept relevance, bypassing of concepts due to linear mappings, lower accuracy compared to black-box models, and a lack of systematic investigation into the impact of visual backbones and vision-language models (VLMs). To overcome these issues, the authors propose CBM-Suite: a framework that introduces an entropy-based measure of concept suitability to quantify concept set effectiveness, inserts a nonlinear layer between concept activations and the classifier to enforce concept-dependent decisions, employs knowledge distillation guided by a linear teacher probe to narrow the performance gap, and systematically evaluates the influence of diverse visual encoders and VLMs on CBM performance. This approach significantly improves both accuracy and interpretability, establishing a new paradigm for reliable concept-driven modeling.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts but face fundamental limitations: the absence of a metric to pre-evaluate concept relevance, the"linearity problem"causing recent CBMs to bypass the concept bottleneck entirely, an accuracy gap compared to opaque models, and finally the lack of systematic study on the impact of different visual backbones and VLMs. We introduce CBM-Suite, a methodological framework to systematically addresses these challenges. First, we propose an entropy-based metric to quantify the intrinsic suitability of a concept set for a given dataset. Second, we resolve the linearity problem by inserting a non-linear layer between concept activations and the classifier, which ensures that model accuracy faithfully reflects concept relevance. Third, we narrow the accuracy gap by leveraging a distillation loss guided by a linear teacher probe. Finally, we provide comprehensive analyses on how different vision encoders, vision-language models, and concept sets interact to influence accuracy and interpretability in CBMs. Extensive evaluations show that CBM-Suite yields more accurate models and provides insights for improving concept-based interpretability.

Problem

• Concept Bottleneck Models
• concept relevance
• linearity problem
• accuracy gap
• vision-language models

Innovation

• Concept Bottleneck Models
• entropy-based metric
• non-linear layer
• distillation loss
• vision-language models