What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

📅 2026-08-25
📈 Citations: 0
Influential: 0
📄 PDF
🤖 AI Summary
研究使用医学视觉-语言模型解决跨域分布偏移导致的性能下降问题,通过不同数据集和方法评估模型的迁移学习、多模态对齐及信息泄露情况。
📝 Abstract
Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.
Problem

Research questions and friction points this paper is trying to address.

Medical Vision-Language Models
Radiology
Distribution Shift
Transfer Failure
Alignment
Innovation

Methods, ideas, or system contributions that make the work stand out.

Medical Vision-Language Models
Cross-Dataset Transfer
Multimodal Alignment
Source-Proxy Leakage
A
Ayoub Louaye Bouaziz
LaTIM, Inserm, University of Western Brittany, Brest, France
L
Lokmane Chebouba
University of Constantine, Algeria
Y
Yassine Himeur
University of Dubai, UAE