Flexible Concept Bottleneck Model

2025-11-10

AI Summary

Existing concept bottleneck models (CBMs) rely on vision-language models (VLMs) but require full-model retraining to incorporate novel concepts, severely limiting adaptability and interpretability in open-world settings. To address this, we propose FlexCBM—a flexible CBM enabling dynamic integration of unseen concepts without retraining. Its core innovations include: (1) a hypernetwork that generates classifier weights in real time conditioned on concept embeddings; (2) a learnable temperature-parameterized sparse attention mechanism for adaptive, sparse concept selection; and (3) support for online replacement and expansion of the concept set. Evaluated on five benchmarks, FlexCBM achieves state-of-the-art accuracy while generalizing to unseen concepts after only a single fine-tuning step. This significantly enhances practical utility, model interpretability, and deployment efficiency—enabling robust, scalable, and transparent concept-based reasoning in dynamic environments.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Concept bottleneck models (CBMs) improve neural network interpretability by introducing an intermediate layer that maps human-understandable concepts to predictions. Recent work has explored the use of vision-language models (VLMs) to automate concept selection and annotation. However, existing VLM-based CBMs typically require full model retraining when new concepts are involved, which limits their adaptability and flexibility in real-world scenarios, especially considering the rapid evolution of vision-language foundation models. To address these issues, we propose Flexible Concept Bottleneck Model (FCBM), which supports dynamic concept adaptation, including complete replacement of the original concept set. Specifically, we design a hypernetwork that generates prediction weights based on concept embeddings, allowing seamless integration of new concepts without retraining the entire model. In addition, we introduce a modified sparsemax module with a learnable temperature parameter that dynamically selects the most relevant concepts, enabling the model to focus on the most informative features. Extensive experiments on five public benchmarks demonstrate that our method achieves accuracy comparable to state-of-the-art baselines with a similar number of effective concepts. Moreover, the model generalizes well to unseen concepts with just a single epoch of fine-tuning, demonstrating its strong adaptability and flexibility.

Problem

• Enables dynamic concept adaptation without full model retraining
• Automates relevant concept selection using learnable temperature parameters
• Maintains accuracy while supporting complete concept set replacement

Innovation

• Hypernetwork generates weights from concept embeddings
• Sparsemax module dynamically selects relevant concepts
• Enables concept replacement without full model retraining