A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

2026-04-04

AI Summary

Spatial transcriptomics remains costly and low-throughput, while hematoxylin and eosin (H&E) staining lacks molecular resolution, hindering precise biological discovery and clinical prediction. To address these limitations, this work proposes STORM, a multimodal foundation model that establishes the first cross-platform, scalable framework for spatial multimodal integration. STORM leverages a hierarchical neural network to jointly encode morphological, gene expression, and spatial contextual information from 1.2 million paired spatial transcriptomics and H&E image patches across 18 organs. The model unifies tissue morphology with molecular representations, significantly outperforming existing methods in gene expression prediction across 11 cancer types. Furthermore, STORM enhances performance in immunotherapy response and prognostic assessment across 23 independent cohorts comprising 7,245 patients, demonstrating compatibility with multiple platforms including Visium, Xenium, Visium HD, and CosMx.

Abstract

Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H\&E) staining offers rich morphology yet lacks molecular resolution. We present \textbf{\ours} (\textbf{S}patial \textbf{T}ranscriptomics and hist\textbf{O}logy \textbf{R}epresentation \textbf{M}odel), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H\&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.

Problem

• Spatial Transcriptomics
• Histology
• Multimodal Integration
• Molecular–Morphological Representation
• Clinical Prediction

Innovation

• foundation model
• spatial transcriptomics
• multimodal integration
• H&E-to-gene expression prediction
• clinical prognostication