From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images

2026-01-25

AI Summary

This work addresses the challenges SAM faces in medical image segmentation due to domain shift, scarce annotations, and its inability to effectively leverage unlabeled data. To overcome these limitations, the authors propose SC-SAM, a novel framework that introduces a bidirectional collaborative training paradigm between a specialist model (U-Net) and the generalist SAM. Specifically, the U-Net generates point prompts and pseudo-labels to guide SAM, while SAM, in turn, imposes regularization constraints on the U-Net, enabling efficient semi-supervised learning from unlabeled data. This approach transcends the constraints of existing parameter-efficient fine-tuning methods, which cannot utilize unlabeled samples, and achieves state-of-the-art performance on benchmark datasets for prostate MRI and polyp segmentation, outperforming current semi-supervised SAM variants and medical foundation models such as MedSAM.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist-generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM.

Problem

• medical image segmentation
• unlabeled data
• domain shift
• semi-supervised learning
• foundation models

Innovation

• specialist-generalist framework
• bidirectional co-training
• semi-supervised medical segmentation
• pseudo-labeling
• Segment Anything Model (SAM)