Leveraging Pathology Foundation Models for Panoptic Segmentation of Melanoma in H&E Images

2025-07-18

AI Summary

Manual segmentation of melanoma tissue in H&E-stained histopathological slides is time-consuming and suffers from poor reproducibility. To address this, we propose an automated pan-tissue segmentation method for five tissue classes, integrating the large-scale pretrained pathology foundation model Virchow2 with an Efficient-UNet architecture. This is the first work to employ Virchow2 as a feature encoder, synergistically fusing its high-level semantic representations with raw RGB spatial information within an encoder-decoder framework—thereby significantly enhancing discriminative capability for complex tissue morphologies. Our method achieves top performance on the PUMA Grand Challenge tissue segmentation benchmark, demonstrating superior generalizability and robustness across diverse datasets. The approach provides a reliable, scalable, and fully automated solution for computational pathology applications, particularly in precision prognostic assessment and treatment planning.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Melanoma is an aggressive form of skin cancer with rapid progression and high metastatic potential. Accurate characterisation of tissue morphology in melanoma is crucial for prognosis and treatment planning. However, manual segmentation of tissue regions from haematoxylin and eosin (H&E) stained whole-slide images (WSIs) is labour-intensive and prone to inter-observer variability, this motivates the need for reliable automated tissue segmentation methods. In this study, we propose a novel deep learning network for the segmentation of five tissue classes in melanoma H&E images. Our approach leverages Virchow2, a pathology foundation model trained on 3.1 million histopathology images as a feature extractor. These features are fused with the original RGB images and subsequently processed by an encoder-decoder segmentation network (Efficient-UNet) to produce accurate segmentation maps. The proposed model achieved first place in the tissue segmentation task of the PUMA Grand Challenge, demonstrating robust performance and generalizability. Our results show the potential and efficacy of incorporating pathology foundation models into segmentation networks to accelerate computational pathology workflows.

Problem

• Automated segmentation of melanoma tissue in H&E images
• Reducing manual labor and variability in tissue analysis
• Improving accuracy in melanoma prognosis and treatment

Innovation

• Leverages Virchow2 foundation model
• Fuses features with RGB images
• Uses Efficient-UNet for segmentation