Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction

2026-06-29

AI Summary

This study addresses the challenge of predicting molecular pathway activity using only routine hematoxylin and eosin (H&E)–stained whole-slide images, thereby circumventing reliance on costly and scarce RNA sequencing data. The work proposes an open-vocabulary molecular prompting framework that leverages frozen, pretrained foundation models for both histopathology images and RNA-seq, integrating a lightweight multimodal alignment module and gene-set semantic prompts to infer pathway activity without end-to-end retraining. This approach enables, for the first time, accurate prediction of diverse molecular pathways from H&E images alone, achieving R² > 0.5 for cell cycle and immune pathways across a multi-cancer cohort of 1,720 cases, improving retrieval performance by 25-fold over baseline methods, and demonstrating clinical relevance and cross-cohort generalizability in the POSEIDON clinical trial.

Abstract

H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting -- querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer cohort (N=1,720), we achieve a 25-fold improvement in retrieval over baseline methods. Systematic analysis reveals a graduated predictability spectrum: morphologically grounded programs (cell-cycle programs, immune-related) are most reliably predicted (R^2>0.5), while predicting pathways with no morphological footprint remains challenging as expected. We validate clinical utility on the POSEIDON clinical trial: H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity and predicted IFN-gamma mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images.

Problem

• Histopathology
• Molecular Prediction
• Multimodal Alignment
• H&E-stained Images
• Transcriptomics

Innovation

• multimodal alignment
• foundation models
• open-vocabulary prompting
• data-efficient learning
• histopathology-based molecular prediction