PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction

2025-09-24

AI Summary

This work addresses the challenges of strong heterogeneity, scale imbalance, and difficult fusion among three clinically relevant modalities—pathology reports, whole-slide histopathology images, and biological pathways (transcriptomic data)—in cancer survival prediction. Methodologically: (i) self-attention mechanisms extract key diagnostic passages from pathology text to construct diagnostic prototypes; (ii) histological and pathway prototypes are generated via clustering to achieve cross-modal representation balance; and (iii) a tri-modal interaction Transformer jointly models intra- and inter-modal dependencies. The key contribution is the first systematic integration of these three clinical-omics modalities for survival prediction, effectively mitigating heterogeneity and imbalance. Evaluated on six TCGA cancer cohorts, the proposed prototypical multimodal Transformer significantly outperforms state-of-the-art methods, demonstrating strong potential for clinical translation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Current multimodal fusion approaches in computational oncology primarily focus on integrating multi-gigapixel histology whole slide images (WSIs) with genomic or transcriptomic data, demonstrating improved survival prediction. We hypothesize that incorporating pathology reports can further enhance prognostic performance. Pathology reports, as essential components of clinical workflows, offer readily available complementary information by summarizing histopathological findings and integrating expert interpretations and clinical context. However, fusing these modalities poses challenges due to their heterogeneous nature. WSIs are high-dimensional, each containing several billion pixels, whereas pathology reports consist of concise text summaries of varying lengths, leading to potential modality imbalance. To address this, we propose a prototype-based approach to generate balanced representations, which are then integrated using a Transformer-based fusion model for survival prediction that we term PS3 (Predicting Survival from Three Modalities). Specifically, we present: (1) Diagnostic prototypes from pathology reports, leveraging self-attention to extract diagnostically relevant sections and standardize text representation; (2) Histological prototypes to compactly represent key morphological patterns in WSIs; and (3) Biological pathway prototypes to encode transcriptomic expressions, accurately capturing cellular functions. PS3, the three-modal transformer model, processes the resulting prototype-based multimodal tokens and models intra-modal and cross-modal interactions across pathology reports, WSIs and transcriptomic data. The proposed model outperforms state-of-the-art methods when evaluated against clinical, unimodal and multimodal baselines on six datasets from The Cancer Genome Atlas (TCGA). The code is available at: https://github.com/manahilr/PS3.

Problem

• Integrates pathology reports with histology images and transcriptomic data
• Addresses modality imbalance between high-resolution images and concise text reports
• Uses prototype-based representations to enhance cancer survival prediction

Innovation

• Prototype-based multimodal token generation
• Transformer fusion for intra- and cross-modal interactions
• Balancing pathology reports, histology images, transcriptomics