Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

2026-05-07

AI Summary

This work addresses the long-standing lack of a unified evaluation protocol in multimodal domain generalization (MMDG), which has hindered clear attribution of performance gains to algorithmic advances versus experimental discrepancies. To this end, the authors introduce MMDG-Bench, the first comprehensive benchmark encompassing six cross-domain datasets, three tasks, six modality combinations, and nine representative methods, enabling systematic assessment under fair conditions across multiple dimensions—standard accuracy, robustness, missing-modality generalization, and model confidence. Based on 7,402 models trained across 95 tasks, the study reveals that current MMDG approaches offer only marginal improvements over empirical risk minimization (ERM) baselines, with no single method consistently outperforming others; trimodal fusion shows no significant advantage over bimodal settings; and all methods exhibit substantial performance degradation under input corruption or missing modalities, exposing critical weaknesses in generalization robustness and reliability.

Abstract

Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures a reliable assessment of the field's advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection. With 7, 402 neural networks trained in total across 95 unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness.

Problem

• Multimodal Domain Generalization
• evaluation benchmark
• model robustness
• missing modalities
• corruption robustness

Innovation

• Multimodal Domain Generalization
• Benchmarking
• Robustness Evaluation
• Missing Modality
• Trustworthy AI