Deep Survival Analysis in Multimodal Medical Data: A Parametric and Probabilistic Approach with Competing Risks

2025-07-10

AI Summary

Addressing the challenge of multimodal survival prediction in oncology, this paper proposes SAMVAE—the first parametric multimodal deep learning framework integrating competing-risks modeling. SAMVAE jointly encodes six heterogeneous modalities: clinical variables, four molecular omics data types (e.g., genomics, transcriptomics), and histopathological images—via modality-specific encoders mapping into a shared latent space—and couples a variational autoencoder with a parametric continuous-time survival model. Innovatively, it introduces interpretable probabilistic outputs and interactive multimodal visualizations to enhance clinical interpretability. Evaluated on breast cancer and low-grade glioma cohorts, SAMVAE achieves state-of-the-art performance in both single-risk and competing-risks survival prediction tasks. It demonstrates high predictive accuracy, robustness to missing modalities and noise, and strong clinical utility—bridging deep learning efficacy with translational relevance in precision oncology.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Accurate survival prediction is critical in oncology for prognosis and treatment planning. Traditional approaches often rely on a single data modality, limiting their ability to capture the complexity of tumor biology. To address this challenge, we introduce a multimodal deep learning framework for survival analysis capable of modeling both single and competing risks scenarios, evaluating the impact of integrating multiple medical data sources on survival predictions. We propose SAMVAE (Survival Analysis Multimodal Variational Autoencoder), a novel deep learning architecture designed for survival prediction that integrates six data modalities: clinical variables, four molecular profiles, and histopathological images. SAMVAE leverages modality specific encoders to project inputs into a shared latent space, enabling robust survival prediction while preserving modality specific information. Its parametric formulation enables the derivation of clinically meaningful statistics from the output distributions, providing patient-specific insights through interactive multimedia that contribute to more informed clinical decision-making and establish a foundation for interpretable, data-driven survival analysis in oncology. We evaluate SAMVAE on two cancer cohorts breast cancer and lower grade glioma applying tailored preprocessing, dimensionality reduction, and hyperparameter optimization. The results demonstrate the successful integration of multimodal data for both standard survival analysis and competing risks scenarios across different datasets. Our model achieves competitive performance compared to state-of-the-art multimodal survival models. Notably, this is the first parametric multimodal deep learning architecture to incorporate competing risks while modeling continuous time to a specific event, using both tabular and image data.

Problem

• Improving survival prediction in oncology using multimodal data
• Integrating diverse medical data sources for accurate prognosis
• Addressing competing risks in continuous-time survival analysis

Innovation

• Multimodal deep learning for survival analysis
• Integrates six data modalities via SAMVAE
• Parametric approach with competing risks