Segment Anything for Histopathology

2025-02-01

AI Summary

In digital pathology, existing nucleus segmentation methods—such as Segment Anything Model (SAM)—exhibit poor generalization across heterogeneous data distributions and rely heavily on labor-intensive manual annotations. To address this, we propose PathoSAM, the first vision foundation model specifically designed for tissue histopathology images to enable robust, general-purpose nucleus segmentation. Built upon the SAM architecture, PathoSAM incorporates multi-source pathological data fine-tuning, multi-task adaptation (supporting both instance and semantic segmentation), and an interactive prompting mechanism to significantly enhance cross-domain robustness. We further provide a whole-slide segmentation script and an open-source annotation toolkit. On nucleus instance segmentation, PathoSAM achieves new state-of-the-art performance; in semantic segmentation, it surpasses all mainstream methods except CellViT. The model, source code, and full toolchain are publicly released.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Nucleus segmentation is an important analysis task in digital pathology. However, methods for automatic segmentation often struggle with new data from a different distribution, requiring users to manually annotate nuclei and retrain data-specific models. Vision foundation models (VFMs), such as the Segment Anything Model (SAM), offer a more robust alternative for automatic and interactive segmentation. Despite their success in natural images, a foundation model for nucleus segmentation in histopathology is still missing. Initial efforts to adapt SAM have shown some success, but did not yet introduce a comprehensive model for diverse segmentation tasks. To close this gap, we introduce PathoSAM, a VFM for nucleus segmentation, based on training SAM on a diverse dataset. Our extensive experiments show that it is the new state-of-the-art model for automatic and interactive nucleus instance segmentation in histopathology. We also demonstrate how it can be adapted for other segmentation tasks, including semantic nucleus segmentation. For this task, we show that it yields results better than popular methods, while not yet beating the state-of-the-art, CellViT. Our models are open-source and compatible with popular tools for data annotation. We also provide scripts for whole-slide image segmentation. Our code and models are publicly available at https://github.com/computational-cell-analytics/patho-sam.

Problem

• Digital Pathology
• Nuclear Segmentation
• SAM (Segment Anything Model)

Innovation

• PathoSAM
• Segment Anything Model
• Digital Pathology