Efficient Few-Shot Medical Image Analysis via Hierarchical Contrastive Vision-Language Learning

2025-01-16

AI Summary

To address the challenge of few-shot classification in medical imaging—particularly for chest X-ray and breast ultrasound—this paper proposes HiCA, a hierarchical contrastive alignment framework. First, domain-adaptive pretraining enhances the medical representation capability of large vision-language models (LVLMs). Second, a two-stage adaptive fine-tuning strategy is introduced, incorporating a novel hierarchical contrastive alignment mechanism that jointly optimizes vision–language cross-modal alignment at both feature-level and semantic-level. Third, a high-quality medical image–text paired dataset is constructed to support contrastive learning. Evaluated on ChestX-ray and Breast Ultrasound benchmarks, HiCA achieves state-of-the-art performance under both few-shot and zero-shot settings, significantly outperforming existing baselines in accuracy. Moreover, it demonstrates strong robustness, cross-modal interpretability, and promising clinical applicability.

Technology Category

• Computer Vision: Medical and Biological Imaging
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Few-shot learning in medical image classification presents a significant challenge due to the limited availability of annotated data and the complex nature of medical imagery. In this work, we propose Adaptive Vision-Language Fine-tuning with Hierarchical Contrastive Alignment (HiCA), a novel framework that leverages the capabilities of Large Vision-Language Models (LVLMs) for medical image analysis. HiCA introduces a two-stage fine-tuning strategy, combining domain-specific pretraining and hierarchical contrastive learning to align visual and textual representations at multiple levels. We evaluate our approach on two benchmark datasets, Chest X-ray and Breast Ultrasound, achieving state-of-the-art performance in both few-shot and zero-shot settings. Further analyses demonstrate the robustness, generalizability, and interpretability of our method, with substantial improvements in performance compared to existing baselines. Our work highlights the potential of hierarchical contrastive strategies in adapting LVLMs to the unique challenges of medical imaging tasks.

Problem

• Medical Image Classification
• Limited Annotated Data
• Chest X-ray and Breast Ultrasound Analysis

Innovation

• Adaptive Visual-Linguistic Fine-tuning
• Hierarchical Contrastive Alignment
• Medical Image Analysis