Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

2026-06-12

AI Summary

This study addresses the performance degradation in time-to-event prediction caused by modality imbalance and distribution shifts in multimodal clinical data. The authors propose a cross-modal alignment framework built upon domain-specific foundation models to separately encode CT imaging and longitudinal electronic health records (EHR), integrating them in a shared latent space through four strategies: late fusion, contrastive alignment, cross-attention, and co-attention. For the first time, they systematically analyze fusion behavior under modality imbalance and formulate task-aware design principles for multimodal alignment, substantially improving generalization across tasks and institutions. On pulmonary embolism mortality and cardiovascular disease prediction tasks, multimodal fusion improves the concordance index by 1.5–5.4% over unimodal baselines. Contrastive fusion—particularly when combined with CLMBR—demonstrates the most robust performance, while cross-attention and image-guided co-attention achieve optimal internal and external results on specific tasks, respectively.

Abstract

Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift. We introduce a foundation model-driven framework for cross-modal representation alignment between CT imaging and longitudinal EHR data, designed to generalize across tasks and institutions. CT and EHR modalities are encoded independently using domain-specific foundation models and aligned in a shared latent space through four principled fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention. We evaluate two clinically distinct TTE tasks: pulmonary embolism (PE) mortality and cardiovascular disease (CVD) outcomes, on large-scale multi-institutional cohorts (PE: N=3,099 train; 1,098 internal; 435 external; CVD: N=2,951 train; 837 internal; 682 external). Fusion consistently improves concordance index by 1.5-5.4% over unimodal baselines when modalities contribute comparably. Overall, contrastive multimodal fusion, particularly with CLMBR representations, provided the most consistent and statistically robust improvements, especially for PE mortality prediction. For MACE, cross-attention (one-hot) achieved the highest internal performance and image-guided co-attention achieved the best external performance. We therefore introduce a generalizable foundation model-based cross-modal alignment framework and provide the first systematic analysis of fusion behavior under modality imbalance in TTE prediction. Our results establish task-aware multimodal alignment as a necessary design principle for robust generalization and scalable clinical deployment.

Problem

• time-to-event prediction
• multimodal fusion
• modality imbalance
• distribution shift
• cross-modal alignment

Innovation

• cross-modal alignment
• foundation models
• time-to-event prediction
• multimodal fusion
• modality imbalance