PanopMamba: Vision State Space Modeling for Nuclei Panoptic Segmentation

2026-01-23

AI Summary

This work addresses the challenges of panoptic segmentation in histopathological images—particularly small object detection, ambiguous boundaries, and class imbalance—by proposing PanopMamba, the first hybrid encoder-decoder architecture to integrate Mamba into panoptic segmentation. By synergistically combining Mamba with Transformers, the method leverages state space modeling to enhance and dynamically fuse multi-scale features, significantly improving semantic and spatial representation of densely packed and overlapping nuclei. The framework incorporates a multi-scale Mamba backbone and an SSM-based fusion network, alongside tailored evaluation metrics including iPQ, wPQ, and fwPQ. Extensive experiments on MoNuSAC2020 and NuInsSeg demonstrate that PanopMamba consistently outperforms existing approaches across multiple metrics, confirming its superior performance and robustness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Nuclei panoptic segmentation supports cancer diagnostics by integrating both semantic and instance segmentation of different cell types to analyze overall tissue structure and individual nuclei in histopathology images. Major challenges include detecting small objects, handling ambiguous boundaries, and addressing class imbalance. To address these issues, we propose PanopMamba, a novel hybrid encoder-decoder architecture that integrates Mamba and Transformer with additional feature-enhanced fusion via state space modeling. We design a multiscale Mamba backbone and a State Space Model (SSM)-based fusion network to enable efficient long-range perception in pyramid features, thereby extending the pure encoder-decoder framework while facilitating information sharing across multiscale features of nuclei. The proposed SSM-based feature-enhanced fusion integrates pyramid feature networks and dynamic feature enhancement across different spatial scales, enhancing the feature representation of densely overlapping nuclei in both semantic and spatial dimensions. To the best of our knowledge, this is the first Mamba-based approach for panoptic segmentation. Additionally, we introduce alternative evaluation metrics, including image-level Panoptic Quality ($i$PQ), boundary-weighted PQ ($w$PQ), and frequency-weighted PQ ($fw$PQ), which are specifically designed to address the unique challenges of nuclei segmentation and thereby mitigate the potential bias inherent in vanilla PQ. Experimental evaluations on two multiclass nuclei segmentation benchmark datasets, MoNuSAC2020 and NuInsSeg, demonstrate the superiority of PanopMamba for nuclei panoptic segmentation over state-of-the-art methods. Consequently, the robustness of PanopMamba is validated across various metrics, while the distinctiveness of PQ variants is also demonstrated. Code is available at https://github.com/mkang315/PanopMamba.

Problem

• nuclei panoptic segmentation
• small object detection
• ambiguous boundaries
• class imbalance
• histopathology images

Innovation

• Mamba
• State Space Model
• Panoptic Segmentation
• Multiscale Fusion
• Nuclei Segmentation