Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics

2026-06-19

AI Summary

This study addresses the challenge of accurately predicting spatial transcriptomic gene expression from H&E histology images alone by formulating the task as spatial interpolation guided by a sparse set of gene anchors. The authors propose a multimodal masked autoencoder that integrates histological imagery with sparse gene expression data to reconstruct whole-slide gene expression maps. Innovatively, they introduce a biologically informed saliency score combined with a learned ranking strategy to adaptively select the most informative contiguous regions as anchors. A cross-modal joint encoder, enhanced with contrastive learning, aligns visual and genomic features in a shared embedding space. Remarkably, the method achieves state-of-the-art performance under extreme conditions—such as only 10% transcriptomic coverage or even in the complete absence of gene anchors—significantly outperforming existing approaches in both image-guided gene expression prediction and spatial interpolation tasks.

Abstract

The high cost of spatial transcriptomics (ST) has driven extensive studies into predicting gene expression directly from H&E histology images. However, this prediction task faces an inherent limitation, as tissue morphology alone provides insufficient information to fully resolve underlying gene expression. To address this limitation, a recent study leverages partial gene expression to guide the prediction process alongside histology images. Building on this paradigm, we approach the prediction task as a spatial imputation problem, employing a Masked Autoencoder (MAE) to utilize a small fraction of gene expression as genetic anchors for inferring whole-slide gene expression profiles. Specifically, we propose a bio-saliency score and a learning-to-rank strategy to adaptively identify the most informative spots within the tissue. Based on these identified spots, our framework selects contiguous regions as genetic anchors to ensure suitability for real-world ST profiling hardware. To effectively leverage these anchors, we design a cross-modal joint encoder that integrates visual and genetic modalities. By aligning the selected anchors with their corresponding visual features via contrastive learning, the encoder generates robust joint representations to accurately predict gene expression across the whole slide. Notably, our framework consistently surpasses existing methods in both histology-only prediction and spatial imputation, achieving superior accuracy even without genetic anchors and further excelling with as little as 10% transcriptomic coverage. Our code is available at https://github.com/Kyyle2114/CAMMST.

Problem

• spatial transcriptomics
• gene expression prediction
• H&E histology
• spatial imputation
• multi-modal learning

Innovation

• Masked Autoencoder
• Spatial Transcriptomics
• Contrastive Learning
• Multi-modal Integration
• Adaptive Anchoring