DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation

2026-04-27

AI Summary

This work addresses the challenge of adapting the Segment Anything Model (SAM/SAM2), pretrained on natural images, to medical image segmentation—a task hindered by domain shift and reliance on user prompts or extensive fine-tuning. The authors propose DiffuSAM, the first approach to integrate a diffusion prior with a frozen SAM2 backbone. By leveraging a lightweight diffusion model to synthesize mask embeddings directly from SAM2 image features and enforcing cross-slice spatial consistency, DiffuSAM enables prompt-free, source-data-free few-shot segmentation of 3D medical images. Evaluated on BTCV and CHAOS datasets across CT and MRI modalities, the method achieves competitive and efficient performance under both few-shot and source-free unsupervised domain adaptation settings.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts. We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes. Evaluated on the BTCV and CHAOS datasets for CT and MRI under Source-Free Unsupervised Domain Adaptation (SF-UDA) and Few-Shot settings, DiffuSAM achieves competitive performance with efficient training and inference. Code is available upon request from the corresponding author.

Problem

• medical image segmentation
• prompt-free
• source-free
• few-shot
• domain adaptation

Innovation

• Diffusion-based segmentation
• Prompt-free SAM
• Source-Free Unsupervised Domain Adaptation
• Few-Shot Medical Image Segmentation
• Spatial Consistency