MedicoSAM: Towards foundation models for medical image segmentation

2025-01-20

AI Summary

To address annotation scarcity, poor model generalizability, and high task-specific adaptation costs in medical image segmentation, this work presents the first systematic adaptation and optimization of the general-purpose vision foundation model Segment Anything Model (SAM) for clinical-grade medical imaging analysis. We propose MedicoSAM—a domain-specialized variant trained on a multi-center, multi-modal (CT/MRI/histopathology) medical dataset—integrating supervised fine-tuning, prompt engineering, and interactive training strategies. Our empirical analysis reveals that interactive segmentation performance improves substantially with interaction-aware adaptation, whereas semantic segmentation benefits more from domain-specific architectural and data-level adaptation than from generic medical pretraining alone. MedicoSAM is the first open-source, plug-and-play SAM variant tailored for healthcare applications, fully compatible with mainstream annotation tools. It consistently outperforms the original SAM in interactive segmentation tasks and enables rapid deployment for clinical research and practice.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Medical image segmentation is an important analysis task in clinical practice and research. Deep learning has massively advanced the field, but current approaches are mostly based on models trained for a specific task. Training such models or adapting them to a new condition is costly due to the need for (manually) labeled data. The emergence of vision foundation models, especially Segment Anything, offers a path to universal segmentation for medical images, overcoming these issues. Here, we study how to improve Segment Anything for medical images by comparing different finetuning strategies on a large and diverse dataset. We evaluate the finetuned models on a wide range of interactive and (automatic) semantic segmentation tasks. We find that the performance can be clearly improved for interactive segmentation. However, semantic segmentation does not benefit from pretraining on medical images. Our best model, MedicoSAM, is publicly available at https://github.com/computational-cell-analytics/medico-sam. We show that it is compatible with existing tools for data annotation and believe that it will be of great practical value.

Problem

• Medical Image Segmentation
• Generic Model
• Cost Reduction

Innovation

• Medical Image Segmentation
• Segment Anything Model (SAM)
• Generalization Across Image Types