Cell Instance Segmentation: The Devil Is in the Boundaries

2025-10-10

AI Summary

Existing cell instance segmentation methods predominantly rely on pixel-wise regression targets (e.g., distance maps, heatmaps), which struggle to preserve global geometric properties of cells—such as shape, curvature, and convexity. To address this, we propose Ceb, a boundary-aware framework that abandons pixel-wise regression in favor of explicitly modeling foreground–foreground boundaries. Ceb fuses foreground–foreground and background–foreground boundary features via boundary signatures and trains a boundary classifier for precise boundary discrimination. Subsequently, candidate boundaries are extracted from semantic segmentation probability maps using an improved Watershed algorithm, enabling boundary-guided clustering of foreground pixels. Evaluated on six public benchmarks, Ceb significantly outperforms mainstream pixel-clustering approaches and matches the performance of state-of-the-art instance segmentation models. To our knowledge, this is the first work to systematically integrate boundary perception into cell instance segmentation, thereby substantially improving geometric fidelity and segmentation accuracy.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

State-of-the-art (SOTA) methods for cell instance segmentation are based on deep learning (DL) semantic segmentation approaches, focusing on distinguishing foreground pixels from background pixels. In order to identify cell instances from foreground pixels (e.g., pixel clustering), most methods decompose instance information into pixel-wise objectives, such as distances to foreground-background boundaries (distance maps), heat gradients with the center point as heat source (heat diffusion maps), and distances from the center point to foreground-background boundaries with fixed angles (star-shaped polygons). However, pixel-wise objectives may lose significant geometric properties of the cell instances, such as shape, curvature, and convexity, which require a collection of pixels to represent. To address this challenge, we present a novel pixel clustering method, called Ceb (for Cell boundaries), to leverage cell boundary features and labels to divide foreground pixels into cell instances. Starting with probability maps generated from semantic segmentation, Ceb first extracts potential foreground-foreground boundaries with a revised Watershed algorithm. For each boundary candidate, a boundary feature representation (called boundary signature) is constructed by sampling pixels from the current foreground-foreground boundary as well as the neighboring background-foreground boundaries. Next, a boundary classifier is used to predict its binary boundary label based on the corresponding boundary signature. Finally, cell instances are obtained by dividing or merging neighboring regions based on the predicted boundary labels. Extensive experiments on six datasets demonstrate that Ceb outperforms existing pixel clustering methods on semantic segmentation probability maps. Moreover, Ceb achieves highly competitive performance compared to SOTA cell instance segmentation methods.

Problem

• Addressing geometric property loss in cell instance segmentation
• Leveraging boundary features to improve cell instance identification
• Developing pixel clustering method using boundary signatures and labels

Innovation

• Revised Watershed algorithm extracts foreground boundaries
• Boundary signatures sample pixels from adjacent boundaries
• Boundary classifier predicts labels to merge regions