$MV_{Hybrid}$: Improving Spatial Transcriptomics Prediction with Hybrid State Space-Vision Transformer Backbone in Pathology Vision Foundation Models

2025-08-01

AI Summary

Existing Vision Transformer (ViT)-based histopathological foundation models underperform in predicting spatial gene expression (biomarkers), primarily due to their limited capacity to capture low-frequency, subtle morphological features associated with molecular phenotypes. Method: We propose $MV_{Hybrid}$, the first hybrid backbone integrating a state-space model (SSM) — initialized with negative real eigenvalues — into a Vision Transformer architecture to enhance low-frequency signal modeling. Leveraging the DINOv2 self-supervised learning framework, we systematically compare six backbone variants and adopt a leave-one-study-out external validation strategy to rigorously assess generalizability. Results: On cross-study spatial gene expression prediction, $MV_{Hybrid}$ achieves a 57% higher correlation and 43% lower performance degradation compared to the best-performing ViT baseline. Moreover, it consistently outperforms existing models across diverse downstream tasks—including classification, retrieval, and survival prediction—demonstrating superior robustness and transferability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Spatial transcriptomics reveals gene expression patterns within tissue context, enabling precision oncology applications such as treatment response prediction, but its high cost and technical complexity limit clinical adoption. Predicting spatial gene expression (biomarkers) from routine histopathology images offers a practical alternative, yet current vision foundation models (VFMs) in pathology based on Vision Transformer (ViT) backbones perform below clinical standards. Given that VFMs are already trained on millions of diverse whole slide images, we hypothesize that architectural innovations beyond ViTs may better capture the low-frequency, subtle morphological patterns correlating with molecular phenotypes. By demonstrating that state space models initialized with negative real eigenvalues exhibit strong low-frequency bias, we introduce $MV_{Hybrid}$, a hybrid backbone architecture combining state space models (SSMs) with ViT. We compare five other different backbone architectures for pathology VFMs, all pretrained on identical colorectal cancer datasets using the DINOv2 self-supervised learning method. We evaluate all pretrained models using both random split and leave-one-study-out (LOSO) settings of the same biomarker dataset. In LOSO evaluation, $MV_{Hybrid}$ achieves 57% higher correlation than the best-performing ViT and shows 43% smaller performance degradation compared to random split in gene expression prediction, demonstrating superior performance and robustness, respectively. Furthermore, $MV_{Hybrid}$ shows equal or better downstream performance in classification, patch retrieval, and survival prediction tasks compared to that of ViT, showing its promise as a next-generation pathology VFM backbone. Our code is publicly available at: https://github.com/deepnoid-ai/MVHybrid.

Problem

• Predicting spatial gene expression from pathology images
• Improving Vision Transformer models for clinical standards
• Combining state space models with ViT for better performance

Innovation

• Hybrid state space-Vision Transformer backbone
• State space models with negative real eigenvalues
• DINOv2 self-supervised learning method