MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

2026-01-31

AI Summary

This work addresses the significant performance degradation in multimodal affective computing under real-world conditions due to missing modalities and the absence of a unified, fair benchmark for robustness evaluation. To this end, we propose MissMAC-Bench, the first standardized evaluation framework specifically designed for modality-missing scenarios. It introduces both fixed and random missing patterns and adheres to two key design principles: training without prior knowledge of missing modalities and employing a single model capable of handling both complete and incomplete inputs. Built upon cross-modal collaboration mechanisms, the benchmark systematically evaluates three prominent language models and various multimodal approaches across four widely used datasets, establishing a reliable and reproducible platform for advancing robust multimodal affective computing.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As a knowledge discovery task over heterogeneous data sources, current Multimodal Affective Computing (MAC) heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce MissMAC-Bench, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust multimodal affective computing and promotes the development of multimedia data mining.

Problem

• missing modality
• multimodal affective computing
• robustness
• distribution shift
• semantic deficiency

Innovation

• missing modality
• multimodal affective computing
• benchmark
• cross-modal synergy
• robustness