impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction

2025-08-08

AI Summary

In cancer survival prediction, the pervasive incompleteness of multimodal data—such as genomics, medical imaging, and clinical records—severely degrades model performance and hinders mechanistic interpretability. To address this, we propose impuTMAE, an end-to-end Transformer-based model. Its core innovation lies in jointly modeling cross-modal interactions and performing missing modality imputation during masked autoencoding pretraining, enabling robust joint learning over heterogeneous, incomplete multimodal inputs. impuTMAE uniformly integrates DNA methylation, RNA-seq, MRI, whole-slide images (WSI), and clinical variables. Evaluated on TCGA-GBM/LGG and BraTS datasets, it achieves state-of-the-art performance in glioma survival prediction, with markedly improved robustness and generalizability across diverse missingness patterns. Moreover, its architecture facilitates post-hoc interpretability, offering a novel paradigm for explainable prognostic modeling under multimodal data incompleteness.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The use of diverse modalities, such as omics, medical images, and clinical data can not only improve the performance of prognostic models but also deepen an understanding of disease mechanisms and facilitate the development of novel treatment approaches. However, medical data are complex, often incomplete, and contains missing modalities, making effective handling its crucial for training multimodal models. We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy. It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches. Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets, integrating five modalities: genetic (DNAm, RNA-seq), imaging (MRI, WSI), and clinical data. By addressing missing data during pre-training and enabling efficient resource utilization, impuTMAE surpasses prior multimodal approaches, achieving state-of-the-art performance in glioma patient survival prediction. Our code is available at https://github.com/maryjis/mtcp

Problem

• Imputes missing modalities in cancer survival prediction
• Handles incomplete multi-modal medical data effectively
• Improves glioma survival prediction performance

Innovation

• Transformer-based end-to-end missing modalities imputation
• Masked pre-training for inter-intra modal interactions
• Heterogeneous data integration for survival prediction