🤖 AI Summary
This work addresses the limitations of early fusion—lacking modularity—and late fusion—neglecting cross-modal interactions—in multimodal sentiment recognition by proposing xgaf, an adaptive fusion method grounded in TreeSHAP attribution. xgaf employs a tree-based mixture-of-experts architecture to dynamically weight unimodal and cross-modal experts. Through systematic evaluation of various SHAP reduction strategies, the study identifies sum-abs as particularly effective, as it preserves total attribution magnitude while enhancing performance. The primary performance gain stems from incorporating trimodal experts rather than from complex routing mechanisms. On the MELD and CMU-MOSEI datasets, xgaf achieves weighted F1 scores of 0.5983 and 0.6519, respectively—significantly outperforming late fusion and matching or slightly surpassing early fusion—while simultaneously maintaining modularity and effectively modeling cross-modal interactions.
📝 Abstract
Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities. In this setting, mean-abs and median-abs reductions can suppress high-dimensional cross-modal experts, whereas sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs \xgaf{} nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 \wf{}, compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs \xgaf{} and early fusion on MELD ($p=1.000$), while \xgaf{} remains significantly better than late fusion ($p<0.0001$). On CMU-MOSEI 3-class sentiment recognition, sum-abs \xgaf{} reaches 0.6519 \wf{}, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show that the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics further show that mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert. Thus, the main contribution is a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion.