Revisiting foundation models for cell instance segmentation

2026-03-18

AI Summary

This study addresses the challenge of instance segmentation of cells, nuclei, and organoids in microscopy images by systematically evaluating the performance of general-purpose and cell-specific foundation models—including the SAM family and its biomedical adaptations such as CellPoseSAM and μSAM—on diverse biomedical imaging datasets. To enhance model adaptability without manual annotation, the authors propose an Automatic Prompt Generation (APG) strategy that optimizes the prompting mechanism of SAM-based architectures. Experimental results demonstrate that APG significantly improves the segmentation accuracy of μSAM across multiple microscopy image datasets, achieving performance comparable to the current state-of-the-art method, CellPoseSAM. This work establishes a new paradigm for the efficient transfer of SAM-based models to biomedical image analysis tasks.

Technology Category

• Computer Vision: Medical and Biological Imaging
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM, $μ$SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for $μ$SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Problem

• cell instance segmentation
• foundation models
• microscopy image analysis
• Segment Anything Model
• model adaptation

Innovation

• Automatic Prompt Generation
• Foundation Models
• Cell Instance Segmentation
• Microscopy Image Analysis
• Segment Anything Model