Conformal Calibration for Multi-Modal Regression with Missing Modalities

2026-08-07

AI Summary

This work addresses the challenge of calibrating prediction intervals in multimodal regression, where missing modalities or inconsistent predictions often undermine reliability. The authors propose a modality-aware conformal calibration layer that equips each modality with an independent predictor and constructs a disagreement score based on their predictive discrepancies. By integrating split conformal prediction with Mondrian stratified calibration under a strict data-splitting protocol, the method adaptively adjusts prediction intervals according to the observed modality pattern at test time. Evaluated across four datasets in 60 experiments, the approach achieves CRPS performance superior or comparable to baselines in 59 cases and yields narrower intervals in 52, while consistently maintaining coverage near 95%. Notably, under modality-missing conditions, it recovers coverage by up to 19.5 percentage points.

Abstract

Prediction intervals for multi-modal regression with tabular variables, text, images, or other input sources are difficult to calibrate when those sources disagree or one is missing. A single global quantile averages these regimes together instead of calibrating to the modality pattern observed at test time. We address this through a modality-aware conformal calibration layer. The layer trains or reuses one predictor per modality, computes a disagreement score from their predictions, and uses that score in split conformal calibration under a strict split protocol. We use the score in two complementary ways. First, a continuous disagreement-scaled method reallocates interval width across examples while preserving the usual marginal split-conformal guarantee. Second, a Mondrian (stratified) method calibrates within groups defined by disagreement or modality availability fixed before calibration, giving group guarantees under joint exchangeability of the calibration and test examples. Across four multi-modal datasets, the disagreement-scaled layer matches or improves the marginal conformal baseline in 59 of 60 paired runs for interval continuous ranked probability score (CRPS) and in 52 of 60 for interval width, while keeping empirical coverage near the 95% target. In stress tests with missing modalities, mask-matched recalibration recovers up to 19.5 percentage points of coverage in the hardest fixed-mask regime. The result is a simple, model-agnostic reliability layer for multi-modal regression systems. A project page is available at https://unco3892.github.io/modality-aware-conformal.

Problem

• conformal calibration
• multi-modal regression
• missing modalities
• prediction intervals
• modality disagreement

Innovation

• conformal calibration
• multi-modal regression
• missing modalities
• disagreement score
• Mondrian calibration