Concept-wise Attention for Fine-grained Concept Bottleneck Models

2026-04-17

AI Summary

This work addresses the limitations of existing concept bottleneck models, which suffer from pretraining biases—such as mismatched granularity and reliance on structural priors—and fail to account for mutual exclusivity among concepts due to the use of binary cross-entropy loss, resulting in suboptimal image–concept alignment. To overcome these issues, the authors propose CoAt-CBM, a novel framework that introduces learnable concept-level visual queries and a concept contrastive optimization mechanism. Built upon CLIP, CoAt-CBM adaptively extracts fine-grained visual embeddings and explicitly models both the relative importance and mutual exclusivity among concepts. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art approaches across multiple benchmarks, achieving notable improvements in concept prediction accuracy and image–concept alignment quality.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recently impressive performance has been achieved in Concept Bottleneck Models (CBM) by utilizing the image-text alignment learned by a large pre-trained vision-language model (i.e. CLIP). However, there exist two key limitations in concept modeling. Existing methods often suffer from pre-training biases, manifested as granularity misalignment or reliance on structural priors. Moreover, fine-tuning with Binary Cross-Entropy (BCE) loss treats each concept independently, which ignores mutual exclusivity among concepts, leading to suboptimal alignment. To address these limitations, we propose Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM), a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability. Specifically, CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector. Then, a novel concept contrastive optimization guides the model to handle the relative importance of the concept scores, enabling concept predictions to faithfully reflect the image content and improved alignment. Extensive experiments demonstrate that CoAt-CBM consistently outperforms state-of-the-art methods. The codes will be available upon acceptance.

Problem

• Concept Bottleneck Models
• pre-training bias
• granularity misalignment
• mutual exclusivity
• image-concept alignment

Innovation

• Concept Bottleneck Models
• Concept-wise Attention
• Fine-grained Alignment
• Concept Contrastive Optimization
• Vision-Language Models