Bridging visual saliency and large language models for explainable deep learning in medical imaging

2026-05-07

AI Summary

This study addresses the limited clinical trustworthiness of deep learning models in medical imaging due to their "black-box" nature. To enhance interpretability, the authors propose a multimodal explainable framework that employs a dual-output CNN for brain tumor classification and integrates Grad-CAM++ to generate visual saliency maps. For the first time, these saliency maps are mapped onto anatomical structures using the Harvard-Oxford cortical atlas and subsequently fed as structured inputs into large language models—such as Grok3 and LLaMA—to automatically produce clinically readable radiology reports. Evaluation on 4,834 brain MRI cases demonstrates that InceptionResNetV2 achieves the best classification performance, Grad-CAM++ yields the highest segmentation accuracy, Grok3 generates the most lexically diverse reports, and LLaMA produces the most readable narratives, thereby unifying technical traceability with semantic interpretability in diagnostic outcomes.

Abstract

The opaque nature of deep learning models remains a significant barrier to their clinical adoption in medical imaging. This paper presents a multimodal explainability framework that bridges the gap between convolutional neural network (CNN) predictions and clinically actionable insights for brain tumor classification, leveraging large language models (LLMs) to deliver human-interpretable diagnostic narratives. The proposed framework operates through three coupled stages. First, nine CNN architectures are extended with a dual-output hybrid formulation that simultaneously optimises a classification head and a segmentation head, enabling spatially richer feature learning. Second, visual saliency attribution methods, namely Grad-CAM, Grad-CAM++, and ScoreCAM, are applied to generate class-discriminative heatmaps, which are subsequently refined into binary tumor masks via an adaptive percentile thresholding pipeline. Third, the resulting masks are mapped onto the Harvard-Oxford cortical atlas to translate pixel-level evidence into named neuroanatomical structures, and the extracted findings are encoded into a structured JSON file that conditions three LLMs (Grok3, Mistral, and LLaMA) to generate coherent, radiological-style diagnostic reports. Evaluated on a dataset of 4,834 contrast-enhanced T1-weighted brain MRI images spanning three tumor classes, InceptionResNetV2 achieved the highest classification performance and Grad-CAM++ yielded the best segmentation overlap. Among the language models, Grok3 led in lexical diversity and coherence, while LLaMA achieved the highest readability score. By integrating visual, anatomical, and linguistic modalities into a unified pipeline, the framework produces explanations that are technically grounded and meaningfully interpretable, advancing the transparency and clinical accountability of artificial intelligence assisted brain tumor diagnosis.

Problem

• explainable AI
• medical imaging
• visual saliency
• large language models
• brain tumor classification

Innovation

• multimodal explainability
• visual saliency
• large language models
• neuroanatomical mapping
• dual-output CNN