🤖 AI Summary
This work addresses the problem of miscalibrated confidence in multimodal fusion caused by missing modalities. It proposes Modal-Conditional Conformal Fusion (MCCF), a method that integrates evidential deep learning with Dempster–Shafer theory. During training, MCCF simulates missing modalities via random modality dropout, allowing absent modalities to contribute vacuous evidence automatically. By incorporating Mondrian conformal prediction, MCCF provides finite-sample coverage guarantees for any non-empty subset of available modalities at test time—without requiring imputation. To the best of our knowledge, MCCF is the first approach to achieve formally calibrated uncertainty under arbitrary modality availability, while also decomposing evidence to yield modality-level nullity scores for uncertainty attribution. Experiments on synthetic data and three real-world benchmarks demonstrate that MCCF consistently attains target coverage, substantially narrows the coverage gap between full and partial modalities, and preserves predictive accuracy.
📝 Abstract
Multimodal fusion architectures typically assume all modalities are available at inference, yet sensor failures, acquisition variability, and cost constraints routinely produce incomplete observations. Existing work treats modality absence as a prediction-accuracy problem, leaving a more basic question unanswered: whether a model's confidence estimates remain calibrated when an entire input stream is removed. We argue that missing-modality robustness and calibrated uncertainty are a single coupled property, and introduce Modality-Conditioned Conformal Fusion (MCCF), an architecture that addresses both at once. MCCF combines a multimodal bottleneck fusion backbone trained with modality dropout, per-modality evidential heads producing modality-decomposed Dirichlet distributions, and a Dempster-Shafer combination rule that fuses the per-modality evidence into a joint predictive distribution; an absent modality contributes vacuous evidence that is structurally ignored, so the fused uncertainty automatically reflects the reduced information without test-time imputation. A Mondrian conformal calibration module keyed on the modality-presence mask then provides finite-sample group-conditional coverage for every non-empty modality subset. MCCF is, to our knowledge, the first method with formal coverage guarantees under arbitrary modality availability through architectural integration rather than post-hoc recalibration, and the evidential decomposition yields per-modality vacuity scores that localise uncertainty to the absent modality responsible. Across a synthetic problem and three real multimodal benchmarks, MCCF holds its target coverage on every modality-presence subset, substantially narrows the coverage gap between full and partial modalities relative to a marginal split-conformal baseline, and imposes no measurable accuracy cost relative to temperature-scaled and evidential baselines.