Caption Bottleneck Models

2026-07-01

AI Summary

This work addresses the limitations of traditional concept bottleneck models, which rely on predefined concept sets that often introduce label bias and information leakage, thereby compromising the trade-off between interpretability and performance. The authors propose Caption Bottleneck Models (CaBM), which eschew handcrafted or static concepts in favor of dynamic, task-relevant concept representations derived from free-text image captions generated by large multimodal models. By constructing a purely textual classification pathway that excludes direct access to visual features, CaBM ensures that explanations are free from information leakage. Moreover, the framework enables post-hoc discovery of dataset-specific concepts without external lexicons. Experimental results demonstrate that CaBM achieves state-of-the-art performance on both fine-grained and coarse-grained classification benchmarks while delivering high-quality, self-contained interpretability.

Abstract

Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.

Problem

• Concept Bottleneck Models
• interpretability
• information leakage
• static concept sets
• label bias

Innovation

• Caption Bottleneck Models
• concept discovery
• interpretability
• information leakage
• open-vocabulary