Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

2026-02-18

AI Summary

This work addresses the challenge of effectively leveraging multimodal data, which is often hindered by missing modalities, asynchronous acquisition, or incomplete samples. To this end, the authors propose PRIMO, a novel method that introduces, for the first time in multimodal learning, a supervised latent-variable-based instance-level modality influence metric. By integrating variational inference with joint multimodal modeling, PRIMO fully exploits all available samples during both training and inference while quantifying each modality’s contribution to predictions. Experimental results demonstrate that PRIMO achieves performance on par with baseline methods across diverse tasks—including synthetic XOR, Audio-Visual MNIST, and MIMIC-III—under both full-modality and single-modality settings. Moreover, it enables visualization of how missing modalities affect predictions, thereby supporting robust and interpretable inference.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.

Problem

• multimodal learning
• missing modalities
• predictive impact
• incomplete data
• latent-variable modeling

Innovation

• multimodal learning
• missing modality
• latent-variable modeling
• predictive impact quantification
• supervised imputation