Concept-based explanation of gene expression prediction from H&E images

2026-08-17

AI Summary

This study addresses the limitation of existing Vision Transformer interpretability methods in elucidating how morphological concepts contribute to spatial transcriptomics predictions. We propose a concept graph framework integrating Layer-wise Relevance Propagation with Top-K Sparse Autoencoders to enable global morphology-molecular association analysis from H&E images to gene expression, overcoming the constraints of local heatmaps. Experimental results demonstrate that the model achieves an F1 score of 0.872 in iCMS classification, effectively stratifies patient prognosis, and exhibits robust cross-dataset generalizability. Consequently, this work establishes a novel interpretable paradigm for understanding the intrinsic mechanisms linking tissue morphology to transcriptional programs, providing critical insights into the molecular underpinnings of histopathological features.

Abstract

Recent advances in pathology foundation models have enabled accurate prediction of spatial transcriptomics (ST) from routine H&E images. However, existing explainability methods for vision transformer (ViT)-based models are largely limited to local heatmaps and do not reveal how morphological concepts contribute to ST predictions. Here, we introduce an explainable framework that combines relevance propagation and concept discovery to link transcriptional programs to tissue morphology. We developed a ViT-based framework for virtual ST from H&E images that combines ViT-aware layer-wise relevance propagation with relaxed archetypal TopK sparse autoencoder-based concept discovery. This approach provides both local explanations and global insights into the morphological patterns associated with transcriptional programs. We applied the framework to colorectal cancer ST data from the HEST-1k cohort and evaluated its generalizability in TCGA COAD. Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes. Measured and predicted gene expression profiles reveal substantial spatial heterogeneity of the colorectal cancer subtypes iCMS2 and iCMS3 across a large number of samples. Spatially resolved and aggregated iCMS classification achieve weighted F1 scores of 0.872 and 0.819 (0.770 in TCGA COAD), respectively, and both stratify patient outcome. Beyond prediction, our framework establishes a relevance-based concept atlas linking molecular phenotypes to histopathological representations. Comparison of activation- with relevance-derived concepts demonstrates that relevances provide a more direct link between tissue morphology and downstream predictions. We establish a general strategy for concept-based explanation of spatial prediction, and our framework is readily applicable to a broad range of ViT-based pathology models.

Problem

• Concept-based explanation
• Spatial transcriptomics prediction
• H&E images
• Vision Transformer explainability
• Morphological concepts

Innovation

• Concept-based Explanation
• Layer-wise Relevance Propagation
• Sparse Autoencoder
• Spatial Transcriptomics
• Vision Transformer