Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations

2024-01-25

International Conference on Learning Representations

AI Summary

Existing concept bottleneck models (CBMs) struggle to capture high-order interactions among concepts and cannot quantify the conditional dependence probabilities between concepts and predictions, limiting their interpretability and causal intervention capability. To address these limitations, we propose the Energy-driven Concept Bottleneck Model (ECBM), the first CBM that formulates the concept bottleneck as a differentiable joint energy function over inputs, concepts, and labels. This unified representation explicitly encodes nonlinear concept interactions and cross-level conditional dependencies, thereby overcoming the restrictive independence assumptions and intervention failures inherent in conventional CBMs. Our method integrates neural energy function design, energy decomposition–based conditional probability derivation, contrastive learning, and MCMC-based approximate inference. Evaluated on multiple real-world datasets, ECBM achieves significant improvements over state-of-the-art methods in classification accuracy, while enabling computable concept correction paths and fine-grained probabilistic explanations.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Probabilistic Circuits and Graphical Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing methods, such as concept bottleneck models (CBMs), have been successful in providing concept-based interpretations for black-box deep learning models. They typically work by predicting concepts given the input and then predicting the final class label given the predicted concepts. However, (1) they often fail to capture the high-order, nonlinear interaction between concepts, e.g., correcting a predicted concept (e.g.,"yellow breast") does not help correct highly correlated concepts (e.g.,"yellow belly"), leading to suboptimal final accuracy; (2) they cannot naturally quantify the complex conditional dependencies between different concepts and class labels (e.g., for an image with the class label"Kentucky Warbler"and a concept"black bill", what is the probability that the model correctly predicts another concept"black crown"), therefore failing to provide deeper insight into how a black-box model works. In response to these limitations, we propose Energy-based Concept Bottleneck Models (ECBMs). Our ECBMs use a set of neural networks to define the joint energy of candidate (input, concept, class) tuples. With such a unified interface, prediction, concept correction, and conditional dependency quantification are then represented as conditional probabilities, which are generated by composing different energy functions. Our ECBMs address both limitations of existing CBMs, providing higher accuracy and richer concept interpretations. Empirical results show that our approach outperforms the state-of-the-art on real-world datasets.

Problem

• Conceptual Bottleneck Models
• Complex Feature Relationships
• Quantification of Feature-Prediction Association

Innovation

• Energy-based Model
• Concept Bottleneck
• Interpretability Enhancement