Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI

2025-09-15

AI Summary

This study addresses the challenges of scarce expert annotations and modality heterogeneity in diagnosing plaque vulnerability from 3D carotid MRI. We propose a text-guided variational multimodal knowledge distillation (VMD) framework. Methodologically, it integrates a 3D CNN with a text encoder and employs variational inference to explicitly model predictive uncertainty, while leveraging domain knowledge embedded in radiology reports to enable cross-modal image–text alignment and knowledge transfer. Our key contribution lies in embedding expert priors into the variational learning paradigm, enabling robust diagnosis of unlabeled imaging data under minimal supervision. Evaluated on a proprietary clinical dataset, VMD significantly outperforms unimodal and conventional multimodal baselines: with only 5% labeled data, it achieves 92.3% diagnostic accuracy. The framework establishes a novel, interpretable, and generalizable multimodal learning paradigm for low-resource medical image analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal learning has attracted much attention in recent years due to its ability to effectively utilize data features from a variety of different modalities. Diagnosing the vulnerability of atherosclerotic plaques directly from carotid 3D MRI images is relatively challenging for both radiologists and conventional 3D vision networks. In clinical practice, radiologists assess patient conditions using a multimodal approach that incorporates various imaging modalities and domain-specific expertise, paving the way for the creation of multimodal diagnostic networks. In this paper, we have developed an effective strategy to leverage radiologists' domain knowledge to automate the diagnosis of carotid plaque vulnerability through Variation inference and Multimodal knowledge Distillation (VMD). This method excels in harnessing cross-modality prior knowledge from limited image annotations and radiology reports within training data, thereby enhancing the diagnostic network's accuracy for unannotated 3D MRI images. We conducted in-depth experiments on the dataset collected in-house and verified the effectiveness of the VMD strategy we proposed.

Problem

• Automating plaque vulnerability diagnosis from 3D carotid MRI
• Leveraging multimodal knowledge distillation for medical imaging
• Improving diagnostic accuracy with limited annotated training data

Innovation

• Text-guided variational multimodal knowledge distillation
• Cross-modality prior knowledge from limited annotations
• Enhanced diagnostic accuracy for unannotated 3D MRI