Semi-Supervised Medical Image Segmentation via Knowledge Mining from Large Models

2025-03-10

AI Summary

Medical image segmentation models (e.g., U-Net++) suffer from performance degradation due to scarce annotated data. To address this, we propose an iterative knowledge transfer framework for few-shot medical image segmentation. Our method leverages the zero-shot segmentation capability of general-purpose vision foundation models (e.g., SAM) to generate initial pseudo-labels on unlabeled images; refines these pseudo-labels iteratively via prompt engineering and backward optimization to yield high-quality, domain-adapted annotations; and subsequently guides fine-tuning of a lightweight U-Net++. This enables co-evolution between large and small models, supporting efficient local deployment while balancing accuracy and inference speed. On Kvasir-SEG and COVID-QU-Ex, our approach achieves Dice score improvements of +3.0% and +1.2% using only 75% and 50% of labeled data, respectively—surpassing fully supervised U-Net++ trained on 100% annotations.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large-scale vision models like SAM have extensive visual knowledge, yet their general nature and computational demands limit their use in specialized tasks like medical image segmentation. In contrast, task-specific models such as U-Net++ often underperform due to sparse labeled data. This study introduces a strategic knowledge mining method that leverages SAM's broad understanding to boost the performance of small, locally hosted deep learning models. In our approach, we trained a U-Net++ model on a limited labeled dataset and extend its capabilities by converting SAM's output infered on unlabeled images into prompts. This process not only harnesses SAM's generalized visual knowledge but also iteratively improves SAM's prediction to cater specialized medical segmentation tasks via U-Net++. The mined knowledge, serving as"pseudo labels", enriches the training dataset, enabling the fine-tuning of the local network. Applied to the Kvasir SEG and COVID-QU-Ex datasets which consist of gastrointestinal polyp and lung X-ray images respectively, our proposed method consistently enhanced the segmentation performance on Dice by 3% and 1% respectively over the baseline U-Net++ model, when the same amount of labelled data were used during training (75% and 50% of labelled data). Remarkably, our proposed method surpassed the baseline U-Net++ model even when the latter was trained exclusively on labeled data (100% of labelled data). These results underscore the potential of knowledge mining to overcome data limitations in specialized models by leveraging the broad, albeit general, knowledge of large-scale models like SAM, all while maintaining operational efficiency essential for clinical applications.

Problem

• Leverages SAM's broad knowledge to enhance medical image segmentation.
• Addresses sparse labeled data limitations in task-specific models like U-Net++.
• Improves segmentation performance using pseudo labels from unlabeled images.

Innovation

• Leverages SAM's knowledge for medical segmentation
• Converts SAM outputs into prompts for U-Net++
• Uses pseudo labels to enrich training dataset