Single-shot Star-convex Polygon-based Instance Segmentation for Spatially-correlated Biomedical Objects

2025-04-16

AI Summary

Biomedical images often contain spatially nested or correlated objects (e.g., nuclei inside cells), yet existing instance segmentation methods typically treat each object class independently via multi-stage pipelines, failing to explicitly model semantic containment constraints. Method: We propose HydraStarDist and its variant HSD-WBR—two novel architectures enabling single-pass, multi-class nested instance segmentation. Built upon StarDist, HydraStarDist incorporates a star-convex prior to explicitly encode spatial inclusion relationships; HSD-WBR introduces a novel Within-Boundary Regularization (WBR) layer that enforces strict containment of child instances within parent boundaries. We further define Joint True Positive Rate (JTPR) as a new metric quantifying nesting accuracy. Results: Experiments show our method matches StarDist and Cellpose in IoU_R and AP, while significantly outperforming them in JTPR. It robustly models partial inclusion/exclusion relationships in both fluorescence and brightfield images, with computational efficiency suitable for large-scale analysis.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Biomedical images often contain objects known to be spatially correlated or nested due to their inherent properties, leading to semantic relations. Examples include cell nuclei being nested within eukaryotic cells and colonies growing exclusively within their culture dishes. While these semantic relations bear key importance, detection tasks are often formulated independently, requiring multi-shot analysis pipelines. Importantly, spatial correlation could constitute a fundamental prior facilitating learning of more meaningful representations for tasks like instance segmentation. This knowledge has, thus far, not been utilised by the biomedical computer vision community. We argue that the instance segmentation of two or more categories of objects can be achieved in parallel. We achieve this via two architectures HydraStarDist (HSD) and the novel (HSD-WBR) based on the widely-used StarDist (SD), to take advantage of the star-convexity of our target objects. HSD and HSD-WBR are constructed to be capable of incorporating their interactions as constraints into account. HSD implicitly incorporates spatial correlation priors based on object interaction through a joint encoder. HSD-WBR further enforces the prior in a regularisation layer with the penalty we proposed named Within Boundary Regularisation Penalty (WBR). Both architectures achieve nested instance segmentation in a single shot. We demonstrate their competitiveness based on $IoU_R$ and AP and superiority in a new, task-relevant criteria, Joint TP rate (JTPR) compared to their baseline SD and Cellpose. Our approach can be further modified to capture partial-inclusion/-exclusion in multi-object interactions in fluorescent or brightfield microscopy or digital imaging. Finally, our strategy suggests gains by making this learning single-shot and computationally efficient.

Problem

• Single-shot instance segmentation for spatially-correlated biomedical objects
• Incorporating spatial correlation priors to improve segmentation accuracy
• Joint segmentation of nested objects using star-convex polygon-based models

Innovation

• Single-shot star-convex polygon instance segmentation
• Joint encoder for spatial correlation priors
• Within Boundary Regularisation Penalty layer