Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology

2026-02-15

AI Summary

This work addresses the limited representational capacity of existing multimodal computational pathology methods, which predominantly rely on visual and linguistic signals while lacking molecular specificity and histopathological-level supervision. To overcome this, the authors propose STAMP, a novel framework that introduces spatially resolved transcriptomics as a self-supervised signal, effectively integrating histopathology images with gene expression data. They construct SpaVis-6M, a large-scale multimodal dataset, and develop a spatial transcriptomics encoder, multiscale contrastive alignment, and a cross-scale image-to-gene localization mechanism to enable pixel- and molecular-level fusion. Evaluated across six datasets and four downstream tasks, STAMP substantially outperforms current state-of-the-art methods, demonstrating that spatial molecular supervision is pivotal for enhancing model generalization in computational pathology.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Recent years have witnessed remarkable progress in multimodal learning within computational pathology. Existing models primarily rely on vision and language modalities; however, language alone lacks molecular specificity and offers limited pathological supervision, leading to representational bottlenecks. In this paper, we propose STAMP, a Spatial Transcriptomics-Augmented Multimodal Pathology representation learning framework that integrates spatially-resolved gene expression profiles to enable molecule-guided joint embedding of pathology images and transcriptomic data. Our study shows that self-supervised, gene-guided training provides a robust and task-agnostic signal for learning pathology image representations. Incorporating spatial context and multi-scale information further enhances model performance and generalizability. To support this, we constructed SpaVis-6M, the largest Visium-based spatial transcriptomics dataset to date, and trained a spatially-aware gene encoder on this resource. Leveraging hierarchical multi-scale contrastive alignment and cross-scale patch localization mechanisms, STAMP effectively aligns spatial transcriptomics with pathology images, capturing spatial structure and molecular variation. We validate STAMP across six datasets and four downstream tasks, where it consistently achieves strong performance. These results highlight the value and necessity of integrating spatially resolved molecular supervision for advancing multimodal learning in computational pathology. The code is included in the supplementary materials. The pretrained weights and SpaVis-6M are available at: https://github.com/Hanminghao/STAMP.

Problem

• computational pathology
• multimodal learning
• spatial transcriptomics
• molecular supervision
• pathology image representation

Innovation

• spatial transcriptomics
• multimodal learning
• computational pathology
• self-supervised representation learning
• gene-guided alignment