JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognostication

2026-06-23

AI Summary

Existing approaches struggle to effectively integrate histological morphology with spatial genomic data and model their spatial context, limiting the accuracy of in silico gene expression prediction from H&E images and its clinical prognostic utility. To address this, this work proposes JASPR—a self-supervised deep learning framework that, for the first time, explicitly aligns the spatial contexts of histology and spatial transcriptomics within a self-supervised paradigm. JASPR employs a shared-expert architecture to jointly learn both cross-modal shared representations and modality-specific features, leveraging a cross-modal reconstruction objective to integrate whole-slide H&E images with spatial transcriptomic data. Evaluated on breast cancer datasets, JASPR significantly improves the accuracy of virtual expression prediction for 9,248 genes and demonstrates independent clinical prognostic value.

Abstract

Recent studies have shown that spatial properties of tumors are critical for understanding disease biology and predicting patient outcomes. These spatial properties are increasingly uncovered through complementary modalities: spatial transcriptomics (ST) captures spatially-resolved molecular states, while hematoxylin and eosin-stained whole slide images (HE) reveal tissue morphology. While approaches are emerging to fuse these modalities, effective methods that learn not only joint representations but also incorporate spatial context across modalities are lacking. Here, we present JASPR (Joint Spatial Representation learning), a self-supervised deep learning framework that integrates HE images and ST data through a cross-modal reconstruction objective that incorporates spatial context within HE images and ST profiles. It employs shared modules to capture universal spatial properties across modalities, while modality-specific experts encode features unique to morphological and genomic data. We train and validate JASPR on breast cancer datasets, demonstrating that its learned joint representation substantially improves HE-based prediction of 9,248 genes and provides prognostic value for breast cancer outcomes.

Problem

• spatial transcriptomics
• histology
• multimodal integration
• spatial context
• virtual genomic screening

Innovation

• Joint Spatial Representation
• Cross-modal Reconstruction
• Self-supervised Learning
• Spatial Transcriptomics
• Histopathology