Together, Then Apart: Revisiting Multimodal Survival Analysis via a Min-Max Perspective

2025-11-22

AI Summary

Existing multimodal survival analysis methods over-rely on attention-based alignment, leading to loss of modality-specific information and representation collapse. To address this, we propose a “Alignment–Distinctiveness” collaborative modeling framework. Our method employs a min-max optimization in two stages: semantic alignment (“Together”) and preservation of feature diversity (“Apart”). We innovatively introduce imbalance-aware optimal transport to guide prototype alignment, and integrate modality-specific anchors with contrastive regularization to jointly achieve cross-modal fusion and modality-specific representation retention. Evaluated on five TCGA datasets, our approach significantly outperforms state-of-the-art methods in survival prediction accuracy. Moreover, the learned shared and modality-specific representations exhibit enhanced biological interpretability and improved model robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Integrating heterogeneous modalities such as histopathology and genomics is central to advancing survival analysis, yet most existing methods prioritize cross-modal alignment through attention-based fusion mechanisms, often at the expense of modality-specific characteristics. This overemphasis on alignment leads to representation collapse and reduced diversity. In this work, we revisit multi-modal survival analysis via the dual lens of alignment and distinctiveness, positing that preserving modality-specific structure is as vital as achieving semantic coherence. In this paper, we introduce Together-Then-Apart (TTA), a unified min-max optimization framework that simultaneously models shared and modality-specific representations. The Together stage minimizes semantic discrepancies by aligning embeddings via shared prototypes, guided by an unbalanced optimal transport objective that adaptively highlights informative tokens. The Apart stage maximizes representational diversity through modality anchors and a contrastive regularizer that preserve unique modality information and prevent feature collapse. Extensive experiments on five TCGA benchmarks show that TTA consistently outperforms state-of-the-art methods. Beyond empirical gains, our formulation provides a new theoretical perspective of how alignment and distinctiveness can be jointly achieved in for robust, interpretable, and biologically meaningful multi-modal survival analysis.

Problem

• Integrating histopathology and genomics for survival analysis
• Preventing representation collapse in multimodal fusion
• Balancing cross-modal alignment with modality-specific characteristics

Innovation

• Min-max optimization for shared and specific representations
• Unbalanced optimal transport aligns embeddings adaptively
• Contrastive regularizer preserves modality uniqueness and diversity