Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

2025-04-18

AI Summary

Multimodal pre-trained models suffer significant performance degradation under modality-missing scenarios, and existing reconstruction methods neglect uncertainty estimation, compromising downstream reliability. To address this, we propose SURE—a framework that jointly models modality reconstruction and uncertainty estimation in the latent space. SURE is the first to embed statistical error propagation into deep neural networks; it introduces a novel Pearson correlation-based loss function that explicitly enforces consistency between reconstruction fidelity and downstream task objectives; and it enables architecture-agnostic, scalable, end-to-end joint optimization. Evaluated on sentiment analysis, fine-grained classification, and action recognition, SURE achieves state-of-the-art performance, markedly improving model robustness, generalization, and interpretability under incomplete multimodal inputs.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.

Problem

• Handling missing modalities in multimodal pretraining models
• Addressing unreliability in reconstructed missing modalities
• Improving interpretability and performance with uncertainty estimation

Innovation

• Latent space reconstruction for missing modalities
• Uncertainty estimation via Pearson Correlation loss
• Statistical error propagation in deep networks