Enhancing zero-shot learning in medical imaging: integrating clip with advanced techniques for improved chest x-ray analysis

2025-03-17

AI Summary

To address the challenges of scarce medical image annotations and severe class imbalance, this paper proposes MoCoCLIP—a novel medical CLIP framework integrating the Momentum Contrast (MoCo) mechanism for the first time to enhance visual representation consistency and robustness of cross-modal image-text alignment. The method enables zero-shot multi-label thoracic disease classification from chest X-ray (CXR) images without requiring additional labeled data. Key contributions include: (1) a self-supervised visual encoder that alleviates annotation dependency; and (2) an optimized text-image joint embedding space that improves fine-grained pathological semantic matching. Evaluated on ChestX-ray14, MoCoCLIP achieves a zero-shot mean Average Precision (mAP) 6.5 percentage points higher than CheXZero. On CheXpert, it attains a zero-shot Area Under the Curve (AUC) of 0.750—surpassing the baseline (0.746) and demonstrating significantly improved cross-domain generalization.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Due to the large volume of medical imaging data, advanced AI methodologies are needed to assist radiologists in diagnosing thoracic diseases from chest X-rays (CXRs). Existing deep learning models often require large, labeled datasets, which are scarce in medical imaging due to the time-consuming and expert-driven annotation process. In this paper, we extend the existing approach to enhance zero-shot learning in medical imaging by integrating Contrastive Language-Image Pre-training (CLIP) with Momentum Contrast (MoCo), resulting in our proposed model, MoCoCLIP. Our method addresses challenges posed by class-imbalanced and unlabeled datasets, enabling improved detection of pulmonary pathologies. Experimental results on the NIH ChestXray14 dataset demonstrate that MoCoCLIP outperforms the state-of-the-art CheXZero model, achieving relative improvement of approximately 6.5%. Furthermore, on the CheXpert dataset, MoCoCLIP demonstrates superior zero-shot performance, achieving an average AUC of 0.750 compared to CheXZero with 0.746 AUC, highlighting its enhanced generalization capabilities on unseen data.

Problem

• Improves zero-shot learning for chest X-ray analysis
• Addresses class-imbalanced and unlabeled medical datasets
• Enhances detection of pulmonary pathologies using MoCoCLIP

Innovation

• Integrates CLIP with Momentum Contrast (MoCo)
• Enhances zero-shot learning for chest X-rays
• Improves detection of pulmonary pathologies