Fine-grained Multi-class Nuclei Segmentation with Molecular-empowered All-in-SAM Model

2025-08-21

AI Summary

General-purpose vision foundation models exhibit limited generalizability for fine-grained nuclear segmentation in histopathological images—particularly for subtype-specific identification. To address this, we propose Molecular-empowered All-in-SAM, the first framework integrating molecular prior knowledge into the Segment Anything Model (SAM). Our method introduces a semantic-adaptive SAM adapter for task-aligned adaptation, a molecular-oriented correction learning (MOCL) mechanism to guide segmentation boundaries toward biologically meaningful structures, and a low-annotation-dependency collaborative labeling strategy. Evaluated on multiple public pathology datasets, our approach achieves significant improvements in fine-grained nucleus-level classification and segmentation accuracy (mDice ↑3.2–5.8%), maintains robustness under sparse or noisy annotations, and substantially reduces manual annotation effort. The core contribution lies in a novel molecular-information-driven co-optimization paradigm for vision foundation models, bridging domain-specific biological priors with generic visual representation learning.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Purpose: Recent developments in computational pathology have been driven by advances in Vision Foundation Models, particularly the Segment Anything Model (SAM). This model facilitates nuclei segmentation through two primary methods: prompt-based zero-shot segmentation and the use of cell-specific SAM models for direct segmentation. These approaches enable effective segmentation across a range of nuclei and cells. However, general vision foundation models often face challenges with fine-grained semantic segmentation, such as identifying specific nuclei subtypes or particular cells. Approach: In this paper, we propose the molecular-empowered All-in-SAM Model to advance computational pathology by leveraging the capabilities of vision foundation models. This model incorporates a full-stack approach, focusing on: (1) annotation-engaging lay annotators through molecular-empowered learning to reduce the need for detailed pixel-level annotations, (2) learning-adapting the SAM model to emphasize specific semantics, which utilizes its strong generalizability with SAM adapter, and (3) refinement-enhancing segmentation accuracy by integrating Molecular-Oriented Corrective Learning (MOCL). Results: Experimental results from both in-house and public datasets show that the All-in-SAM model significantly improves cell classification performance, even when faced with varying annotation quality. Conclusions: Our approach not only reduces the workload for annotators but also extends the accessibility of precise biomedical image analysis to resource-limited settings, thereby advancing medical diagnostics and automating pathology image analysis.

Problem

• Segmenting fine-grained nuclei subtypes in pathology images
• Reducing detailed pixel-level annotation requirements
• Improving segmentation accuracy with molecular-empowered learning

Innovation

• Molecular-empowered learning reduces annotation workload
• SAM adapter adapts model for specific semantic emphasis
• Molecular-Oriented Corrective Learning enhances segmentation accuracy