SpaCRD: Multimodal Deep Fusion of Histology and Spatial Transcriptomics for Cancer Region Detection

2026-03-06

AI Summary

Current histology-based methods for cancer region detection are prone to interference from morphological similarities, resulting in high false-positive rates and limited ability to effectively integrate histological and spatial transcriptomic data—particularly suffering from poor generalization across samples and across platforms or batches. To address these challenges, this work proposes SpaCRD, a transfer learning–based multimodal deep fusion framework that introduces a novel category-regularized variational reconstruction–guided bidirectional cross-attention mechanism. This mechanism adaptively integrates histomorphological and gene expression information to uncover their latent co-occurrence patterns. Extensive experiments across 23 paired datasets encompassing diverse cancer types, platforms, and batches demonstrate that SpaCRD significantly outperforms eight state-of-the-art methods in cancer region detection, exhibiting exceptional generalization capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Accurate detection of cancer tissue regions (CTR) enables deeper analysis of the tumor microenvironment and offers crucial insights into treatment response. Traditional CTR detection methods, which typically rely on the rich cellular morphology in histology images, are susceptible to a high rate of false positives due to morphological similarities across different tissue regions. The groundbreaking advances in spatial transcriptomics (ST) provide detailed cellular phenotypes and spatial localization information, offering new opportunities for more accurate cancer region detection. However, current methods are unable to effectively integrate histology images with ST data, especially in the context of cross-sample and cross-platform/batch settings for accomplishing the CTR detection. To address this challenge, we propose SpaCRD, a transfer learning-based method that deeply integrates histology images and ST data to enable reliable CTR detection across diverse samples, platforms, and batches. Once trained on source data, SpaCRD can be readily generalized to accurately detect cancerous regions across samples from different platforms and batches. The core of SpaCRD is a category-regularized variational reconstruction-guided bidirectional cross-attention fusion network, which enables the model to adaptively capture latent co-expression patterns between histological features and gene expression from multiple perspectives. Extensive benchmark analysis on 23 matched histology-ST datasets spanning various disease types, platforms, and batches demonstrates that SpaCRD consistently outperforms existing eight state-of-the-art methods in CTR detection.

Problem

• cancer region detection
• histology
• spatial transcriptomics
• cross-platform integration
• multimodal fusion

Innovation

• multimodal fusion
• spatial transcriptomics
• cross-platform generalization
• bidirectional cross-attention
• transfer learning