SD-FSMIS: Adapting Stable Diffusion for Few-Shot Medical Image Segmentation

2026-04-03

AI Summary

This work addresses the challenges of annotation scarcity and domain shift in few-shot medical image segmentation by introducing, for the first time, the large-scale pre-trained diffusion model Stable Diffusion to this task. The authors propose a support-query interaction (SQI) mechanism and a vision-text conditional translation (VTCT) module to enable effective conditional guidance and cross-domain generalization. The resulting conditional generation framework achieves state-of-the-art or superior segmentation performance under both standard and cross-domain few-shot settings, significantly enhancing the model’s generalization capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Few-Shot Medical Image Segmentation (FSMIS) aims to segment novel object classes in medical images using only minimal annotated examples, addressing the critical challenges of data scarcity and domain shifts prevalent in medical imaging. While Diffusion Models (DM) excel in visual tasks, their potential for FSMIS remains largely unexplored. We propose that the rich visual priors learned by large-scale DMs offer a powerful foundation for a more robust and data-efficient segmentation approach. In this paper, we introduce SD-FSMIS, a novel framework designed to effectively adapt the powerful pre-trained Stable Diffusion (SD) model for the FSMIS task. Our approach repurposes its conditional generative architecture by introducing two key components: a Support-Query Interaction (SQI) and a Visual-to-Textual Condition Translator (VTCT). Specifically, SQI provides a straightforward yet powerful means of adapting SD to the FSMIS paradigm. The VTCT module translates visual cues from the support set into an implicit textual embedding that guides the diffusion model, enabling precise conditioning of the generation process. Extensive experiments demonstrate that SD-FSMIS achieves competitive results compared to state-of-the-art methods in standard settings. Surprisingly, it also demonstrated excellent generalization ability in more challenging cross-domain scenarios. These findings highlight the immense potential of adapting large-scale generative models to advance data-efficient and robust medical image segmentation.

Problem

• Few-Shot Medical Image Segmentation
• Data Scarcity
• Domain Shift
• Medical Image Segmentation

Innovation

• Few-Shot Medical Image Segmentation
• Stable Diffusion
• Diffusion Models
• Support-Query Interaction
• Visual-to-Textual Condition Translator