Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology

2026-06-16

AI Summary

This study addresses the limitations of existing methods that predominantly rely on single-image modalities and struggle to accurately predict key immune biomarkers within the tumor immune microenvironment. The authors propose MixTIME, a novel model that introduces a learnable routing mechanism to dynamically integrate three foundational histopathology models—UNIv2, CONCHv1.5, and STPath—within a multimodal mixture-of-experts (MoE) architecture. This enables pixel- and slide-level prediction of multiplex immunofluorescent protein expression directly from H&E whole-slide images. MixTIME further incorporates a distribution- and trend-aware loss function, facilitating dynamic protein tracking across timepoints and uncovering interaction patterns associated with drug resistance. Evaluated on two datasets across 17 protein biomarkers, the model achieves state-of-the-art performance, significantly enhancing spatial resolution, survival prediction accuracy, and AI-assisted pathology reporting, with validation from multiple institutional pathologists.

Abstract

Predicting immune biomarkers associated with the tumor immune microenvironment (TIME) is critical for advancing precision oncology, yet existing approaches are largely limited to single image modalities and suffer from insufficient resolution and incomplete utilization of complementary clinical and biological information. Here we introduce MixTIME, a multimodal foundation model that leverages a mixture-of-experts (MoE) architecture to integrate pathology foundation models trained across distinct modalities: image only (UNIv2), image text (CONCHv1.5), and image transcriptomic (STPath) representations for pixel-level and slide-level prediction of multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin (HE) whole-slide images. MixTIME employs a learnable router to dynamically weight expert contributions and is trained with a distribution- and tendency-aware loss function. Benchmarked on two datasets of different scales, MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics. The predicted mIF profiles substantially enhance downstream tasks, including spatial domain identification, survival prediction, and AI-assisted pathology report generation validated by expert pathologists from multiple institutes across the world. Furthermore, MixTIME enables longitudinal tracking of protein expression dynamics across clinical time points and reveals protein gene interaction patterns linked to drug resistance and immune suppression in tumor microenvironments. Collectively, MixTIME provides a scalable framework for multimodal biomarker discovery and clinical translation in computational pathology.

Problem

• immune biomarkers
• tumor immune microenvironment
• multimodal integration
• precision oncology
• computational pathology

Innovation

• Mixture-of-Experts
• Multimodal Foundation Model
• Computational Pathology
• Immune Biomarker Prediction
• HE-to-mIF Translation