🤖 AI Summary
Existing radiology multimodal models face three key challenges: weak modeling of low-resolution images, insufficient exploitation of radiology report semantics, and clinically uninterpretable cross-modal attention. This paper proposes a novel vision-language interpretable alignment framework. First, it introduces a local patch–text embedding similarity-based cross-attention mechanism—the first of its kind. Second, it designs a multi-positive contrastive learning strategy to enhance fine-grained semantic modeling of radiology reports. Third, it generates pixel-level cross-modal similarity maps that provide clinically interpretable alignment evidence and enable open-vocabulary semantic segmentation. The method integrates large language model semantic distillation with trainable Transformer layers. On public chest X-ray benchmarks, it achieves state-of-the-art zero-shot classification, localization, and segmentation performance. Extensive experiments demonstrate significant improvements in both clinical interpretability and generalization capability.
📝 Abstract
Recent advancements in multi-modal models have significantly improved vision-language alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning, rely on low-resolution images, and offer limited interpretability in attention mechanisms. To address these challenges, we introduce RadZero, a novel similarity-based cross-attention framework for vision-language alignment in radiology with zero-shot multi-task capability. RadZero leverages large language models to extract minimal semantic sentences from radiology reports and employs a multi-positive contrastive learning strategy to effectively capture relationships between images and multiple relevant textual descriptions. It also utilizes a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, RadZero enables zero-shot inference with similarity probability for classification and pixel-level cross-modal similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, cross-modal similarity map analysis highlights its potential for improving explainability in vision-language alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging.