Enhancing Interpretable Image Classification Through LLM Agents and Conditional Concept Bottleneck Models

2025-06-02

AI Summary

In interpretable image classification, determining the optimal number of concepts adaptively remains challenging, often leading to redundancy or insufficient concept coverage. To address this, we propose the Conditional Concept Bottleneck Model (CoCoBM) coupled with a dynamic LLM-driven agent mechanism. Our key contributions are: (1) a novel dynamic concept bank that enables environment-feedback-driven concept addition and removal; and (2) an editable concept scoring matrix, calibrated in real time by a large language model to quantify concept contribution—overcoming the static, non-adjustable nature of conventional Concept Bottleneck Models (CBMs). Evaluated on six standard benchmarks, CoCoBM achieves a 6.0% absolute improvement in classification accuracy and a 30.2% gain in interpretability metrics over strong baselines, significantly enhancing both discriminative capability and explanation fidelity.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Concept Bottleneck Models (CBMs) decompose image classification into a process governed by interpretable, human-readable concepts. Recent advances in CBMs have used Large Language Models (LLMs) to generate candidate concepts. However, a critical question remains: What is the optimal number of concepts to use? Current concept banks suffer from redundancy or insufficient coverage. To address this issue, we introduce a dynamic, agent-based approach that adjusts the concept bank in response to environmental feedback, optimizing the number of concepts for sufficiency yet concise coverage. Moreover, we propose Conditional Concept Bottleneck Models (CoCoBMs) to overcome the limitations in traditional CBMs' concept scoring mechanisms. It enhances the accuracy of assessing each concept's contribution to classification tasks and feature an editable matrix that allows LLMs to correct concept scores that conflict with their internal knowledge. Our evaluations across 6 datasets show that our method not only improves classification accuracy by 6% but also enhances interpretability assessments by 30%.

Problem

• Optimizing the number of interpretable concepts in image classification
• Improving concept scoring mechanisms in Concept Bottleneck Models
• Enhancing classification accuracy and interpretability through dynamic adjustments

Innovation

• Dynamic agent-based concept bank optimization
• Conditional Concept Bottleneck Models (CoCoBMs)
• LLM-editable concept scoring matrix