SENCA-st: Integrating Spatial Transcriptomics and Histopathology with Cross Attention Shared Encoder for Region Identification in Cancer Pathology

2025-11-11

AI Summary

To address modality imbalance, information loss, and over-smoothing in multimodal integration of spatial transcriptomics and histopathological images, this paper proposes SENCA-st—a novel model featuring a shared encoder and a neighborhood cross-attention mechanism. This design preserves modality-specific characteristics while enhancing discriminability among structurally similar but functionally distinct tissue regions. By mitigating unimodal dependency, SENCA-st enables precise identification of tumor heterogeneity and functional compartments within the tumor microenvironment. Evaluated on multiple cancer datasets, SENCA-st consistently outperforms state-of-the-art methods (e.g., SPACEL, STAlign), achieving improvements of 8.2%–14.6% in segmentation and clustering metrics. Furthermore, the model offers inherent interpretability and strong potential for clinical translation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Spatial transcriptomics is an emerging field that enables the identification of functional regions based on the spatial distribution of gene expression. Integrating this functional information present in transcriptomic data with structural data from histopathology images is an active research area with applications in identifying tumor substructures associated with cancer drug resistance. Current histopathology-spatial-transcriptomic region segmentation methods suffer due to either making spatial transcriptomics prominent by using histopathology features just to assist processing spatial transcriptomics data or using vanilla contrastive learning that make histopathology images prominent due to only promoting common features losing functional information. In both extremes, the model gets either lost in the noise of spatial transcriptomics or overly smoothed, losing essential information. Thus, we propose our novel architecture SENCA-st (Shared Encoder with Neighborhood Cross Attention) that preserves the features of both modalities. More importantly, it emphasizes regions that are structurally similar in histopathology but functionally different on spatial transcriptomics using cross-attention. We demonstrate the superior performance of our model that surpasses state-of-the-art methods in detecting tumor heterogeneity and tumor micro-environment regions, a clinically crucial aspect.

Problem

• Integrating spatial transcriptomics with histopathology for cancer region identification
• Overcoming limitations of existing multimodal segmentation methods in oncology
• Detecting tumor heterogeneity and microenvironment regions with cross-attention

Innovation

• Cross-attention shared encoder integrates histopathology and transcriptomics
• Emphasizes structurally similar but functionally distinct regions
• Preserves both modalities' features to avoid information loss