ESP-MedSAM: Efficient Self-Prompting SAM for Universal Domain-Generalized Medical Image Segmentation

2024-07-19

arXiv.org

AI Summary

Existing medical image segmentation models—such as SAM—suffer from high computational overhead, reliance on manual prompts, severe decoding conflicts, and poor cross-modal and cross-domain generalization. To address these bottlenecks, this work proposes a lightweight, prompt-free, and modality-adaptive unified segmentation framework. Our key contributions are: (1) Multi-Modal Decoupled Knowledge Distillation (MMDKD), which drastically reduces model parameters; (2) Self-Generated Patch-wise Prompt Generation (SPPG), eliminating dependence on manual annotations; and (3) Query-Decoupled Modality-Specific Decoder (QDMD), mitigating decoding conflicts while enhancing modality adaptability. The resulting model has only 4.5% of the parameters of SAM-H, yet achieves state-of-the-art performance across diverse multi-modal and multi-center medical segmentation benchmarks. It demonstrates strong domain generalization and modality-agnostic capability, significantly advancing clinical deployability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The universality of deep neural networks across different modalities and their generalization capabilities to unseen domains play an essential role in medical image segmentation. The recent Segment Anything Model (SAM) has demonstrated its potential in both settings. However, the huge computational costs, demand for manual annotations as prompts and conflict-prone decoding process of SAM degrade its generalizability and applicability in clinical scenarios. To address these issues, we propose an efficient self-prompting SAM for universal domain-generalized medical image segmentation, named ESP-MedSAM. Specifically, we first devise the Multi-Modal Decoupled Knowledge Distillation (MMDKD) strategy to construct a lightweight semi-parameter sharing image encoder that produces discriminative visual features for diverse modalities. Further, we introduce the Self-Patch Prompt Generator (SPPG) to automatically generate high-quality dense prompt embeddings for guiding segmentation decoding. Finally, we design the Query-Decoupled Modality Decoder (QDMD) that leverages a one-to-one strategy to provide an independent decoding channel for every modality. Extensive experiments indicate that ESP-MedSAM outperforms state-of-the-arts in diverse medical imaging segmentation tasks, displaying superior modality universality and generalization capabilities. Especially, ESP-MedSAM uses only 4.5% parameters compared to SAM-H. The source code is available at https://github.com/xq141839/ESP-MedSAM.

Problem

• Reduces computational costs of SAM in medical segmentation
• Eliminates need for manual prompts in medical imaging
• Prevents modality interference in multi-modal segmentation

Innovation

• Lightweight domain-controllable image encoder
• Self-patch prompt generator for embeddings
• Query-decoupled modality decoder