Locality-aware Concept Bottleneck Model

2025-08-20

AI Summary

Concept bottleneck models (CBMs) often predict concept presence without accurately localizing their corresponding visual regions, thereby compromising interpretability and spatial awareness. To address this, we propose Locality-Aware CBM—a novel framework that synergistically integrates foundation model supervision with concept-level prototype learning. Specifically, it leverages foundation models to extract semantically robust concept cues and establishes fine-grained alignment between image local regions and concept prototypes; additionally, region-wise contrastive learning is introduced to enhance localization discriminability. Our method jointly optimizes concept prediction and spatial localization without sacrificing classification accuracy. Extensive experiments demonstrate substantial improvements in concept localization—e.g., +12.3% mAP on CUB and AwA2—while preserving semantic fidelity and spatial precision. This work establishes a new paradigm for interpretable AI that unifies conceptual reasoning with geometric grounding.

Technology Category

• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Concept bottleneck models (CBMs) are inherently interpretable models that make predictions based on human-understandable visual cues, referred to as concepts. As obtaining dense concept annotations with human labeling is demanding and costly, recent approaches utilize foundation models to determine the concepts existing in the images. However, such label-free CBMs often fail to localize concepts in relevant regions, attending to visually unrelated regions when predicting concept presence. To this end, we propose a framework, coined Locality-aware Concept Bottleneck Model (LCBM), which utilizes rich information from foundation models and adopts prototype learning to ensure accurate spatial localization of the concepts. Specifically, we assign one prototype to each concept, promoted to represent a prototypical image feature of that concept. These prototypes are learned by encouraging them to encode similar local regions, leveraging foundation models to assure the relevance of each prototype to its associated concept. Then we use the prototypes to facilitate the learning process of identifying the proper local region from which each concept should be predicted. Experimental results demonstrate that LCBM effectively identifies present concepts in the images and exhibits improved localization while maintaining comparable classification performance.

Problem

• Improves concept localization in interpretable models
• Reduces reliance on irrelevant visual regions
• Ensures accurate spatial concept identification

Innovation

• Leverages foundation models for concept identification
• Uses prototype learning for accurate concept localization
• Assigns one prototype per concept to represent features