VitaminP: cross-modal learning enables whole-cell segmentation from routine histology

2026-04-26

AI Summary

Conventional H&E staining suffers from insufficient cytoplasmic contrast, hindering accurate whole-cell segmentation, while multiplex immunofluorescence—though precise—is limited by high cost and low accessibility. This work proposes VitaminP, a novel framework that establishes the first cross-modal supervision mechanism to transfer molecular-level boundary information from paired multiplex immunofluorescence images to H&E images, enabling high-precision whole-cell segmentation using only routine H&E stains. The study introduces a large-scale segmentation dataset encompassing 34 cancer types and over 7 million annotated cells, demonstrating superior performance over four state-of-the-art methods across 14 public benchmarks and exhibiting strong generalization on internal data from 24 rare cancers. An accompanying open platform, VitaminPScope, supports scalable inference.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Accurate whole-cell and nuclear segmentation is essential for precision pathology and spatial omics, yet routine hematoxylin and eosin (H&E) staining provides limited cytoplasmic contrast, restricting analyses to nuclei. Multiplex immunofluorescence (mIF) facilitates precise whole-cell delineation but remains constrained by cost and accessibility. We introduce VitaminP, a cross-modal learning framework enabling whole cell segmentation from H&E images. By learning from paired H&E-mIF data, VitaminP transfers molecular boundary information from mIF to overcome cytoplasmic contrast in H&E, establishing cross-modal supervision as a general strategy for recovering missing biological structure. We train VitaminP on 14 public datasets covering 34 cancer types and over 7 million instances, integrating publicly available labels with extensive annotations generated in this study, forming one of the largest resources for segmentation. VitaminP outperforms four state-of-the-art methods and generalizes to unseen datasets, including an in-house dataset spanning 24 rare cancer types. We further developed VitaminPScope, an open-source platform providing an interface for scalable inference and enabling broad adoption.

Problem

• whole-cell segmentation
• H&E staining
• cytoplasmic contrast
• multiplex immunofluorescence
• precision pathology

Innovation

• cross-modal learning
• whole-cell segmentation
• H&E staining
• multiplex immunofluorescence
• spatial omics