Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models
🤖 AI Summary
This study investigates the textual concepts learned by deep learning models in mammography to enhance clinical AI interpretability and safety. We propose Mammo-CLIP Dissect—the first concept-level interpretability framework for breast imaging vision models—integrating a breast-specific vision-language model (Mammo-CLIP), neuron-level concept attribution, and semantic alignment analysis to systematically quantify model representations of clinically relevant concepts (e.g., benign calcifications, density features). By comparing models trained on generic versus domain-specific data and analyzing fine-tuning effects, we find that domain-specific pretraining better aligns with radiologists’ diagnostic reasoning, while fine-tuning strengthens representation of critical concepts—albeit at the cost of a trade-off between calcification and density encoding. Our framework enables granular, concept-driven model auditing and reveals previously undocumented representational biases. The implementation code and a curated, clinically validated concept set are publicly released to support reproducible, interpretable mammography AI research.
Application Category
📝 Abstract
Understanding what deep learning (DL) models learn is essential for the safe deployment of artificial intelligence (AI) in clinical settings. While previous work has focused on pixel-based explainability methods, less attention has been paid to the textual concepts learned by these models, which may better reflect the reasoning used by clinicians. We introduce Mammo-CLIP Dissect, the first concept-based explainability framework for systematically dissecting DL vision models trained for mammography. Leveraging a mammography-specific vision-language model (Mammo-CLIP) as a "dissector," our approach labels neurons at specified layers with human-interpretable textual concepts and quantifies their alignment to domain knowledge. Using Mammo-CLIP Dissect, we investigate three key questions: (1) how concept learning differs between DL vision models trained on general image datasets versus mammography-specific datasets; (2) how fine-tuning for downstream mammography tasks affects concept specialisation; and (3) which mammography-relevant concepts remain underrepresented. We show that models trained on mammography data capture more clinically relevant concepts and align more closely with radiologists' workflows than models not trained on mammography data. Fine-tuning for task-specific classification enhances the capture of certain concept categories (e.g., benign calcifications) but can reduce coverage of others (e.g., density-related features), indicating a trade-off between specialisation and generalisation. Our findings show that Mammo-CLIP Dissect provides insights into how convolutional neural networks (CNNs) capture mammography-specific knowledge. By comparing models across training data and fine-tuning regimes, we reveal how domain-specific training and task-specific adaptation shape concept learning. Code and concept set are available: https://github.com/Suaiba/Mammo-CLIP-Dissect.
Problem
Research questions and friction points this paper is trying to address.
Analyzing textual concepts learned by mammography deep learning models
Comparing concept learning between general and mammography-specific trained models
Investigating how fine-tuning affects concept specialization in mammography AI
Innovation
Methods, ideas, or system contributions that make the work stand out.
Leverages mammography-specific vision-language model as dissector
Labels neurons with human-interpretable textual concepts
Quantifies concept alignment with clinical domain knowledge
🔎 Similar Papers
No similar papers found.
