Fusion or Confusion? Multimodal Complexity Is Not All You Need

2025-12-28

AI Summary

This work challenges the prevailing assumption that complex multimodal architectures inherently outperform simpler approaches. To this end, we conduct a large-scale empirical study, systematically reproducing 19 state-of-the-art methods across nine benchmark datasets—spanning up to 23 modalities—under standardized evaluation protocols. We introduce SimBaMM, a lightweight Late-Fusion Transformer baseline, and rigorously assess it using automated hyperparameter optimization, missing-modality robustness evaluation, and statistical significance testing. Our results reveal that most sophisticated models fail to significantly outperform SimBaMM under fair, tuned conditions; in low-data regimes, they often underperform optimized unimodal baselines; and original publications frequently suffer from evaluation bias and irreproducibility. Contributions include: (1) the first reliability-aware evaluation framework for multimodal learning, (2) the SimBaMM baseline, and (3) a multimodal evaluation checklist—collectively establishing a reproducible, comparable benchmarking paradigm for the field.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Deep learning architectures for multimodal learning have increased in complexity, driven by the assumption that multimodal-specific methods improve performance. We challenge this assumption through a large-scale empirical study reimplementing 19 high-impact methods under standardized conditions, evaluating them across nine diverse datasets with up to 23 modalities, and testing their generalizability to new tasks beyond their original scope, including settings with missing modalities. We propose a Simple Baseline for Multimodal Learning (SimBaMM), a straightforward late-fusion Transformer architecture, and demonstrate that under standardized experimental conditions with rigorous hyperparameter tuning of all methods, more complex architectures do not reliably outperform SimBaMM. Statistical analysis indicates that more complex methods perform comparably to SimBaMM and frequently do not reliably outperform well-tuned unimodal baselines, especially in the small-data regime considered in many original studies. To support our findings, we include a case study of a recent multimodal learning method highlighting the methodological shortcomings in the literature. In addition, we provide a pragmatic reliability checklist to promote comparable, robust, and trustworthy future evaluations. In summary, we argue for a shift in focus: away from the pursuit of architectural novelty and toward methodological rigor.

Problem

• Challenges assumption that complex multimodal methods improve performance
• Evaluates 19 methods across diverse datasets and missing modalities
• Proposes simple baseline showing complex architectures not reliably better

Innovation

• Proposed Simple Baseline for Multimodal Learning (SimBaMM)
• Used late-fusion Transformer architecture for multimodal tasks
• Challenged complex methods with standardized empirical evaluation