MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

2025-05-15

AI Summary

This study addresses the clinical scalability limitations of multiplex immunofluorescence (mIF) imaging—namely, high cost and procedural complexity—by proposing a novel end-to-end paradigm to predict multi-target mIF protein expression directly from routine hematoxylin and eosin (H&E) images. Methodologically, we introduce a cross-modal architecture integrating a Vision Transformer (ViT) encoder with a U-Net decoder, augmented with multi-task regression and classification heads. Leveraging transfer learning on the ORION dataset, we achieve the first simultaneous prediction of five mIF biomarkers (Pan-CK, CD3, CD8, FOXP3, α-SMA) from H&E in colorectal cancer tissue. Quantitative evaluation yields F1 scores of 0.88, 0.57, 0.56, 0.36, and 0.30—substantially surpassing state-of-the-art methods. Furthermore, our model uncovers learnable associations between nuclear morphological features and molecular phenotypes, establishing a scalable, H&E-driven framework for constructing high-resolution cellular spatial atlases.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E), a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on two independent datasets. MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that our model effectively captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

Problem

• Predicts multiplex immunofluorescence signals from H&E images
• Enables cell-type identification without costly mIF staining
• Improves cancer diagnosis via H&E-based spatial cellular analysis

Innovation

• U-Net-inspired architecture with ViT encoders
• Predicts mIF signals from H&E images
• Accurate cell-type classification from H&E