Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

2026-04-14

AI Summary

This work addresses the challenge in multimodal sentiment analysis where dominant modalities often suppress weaker ones, leading to degraded fusion performance and insufficient robustness under modality missingness or noise. To mitigate this, the authors propose an enhancement–rebalancing collaborative framework that strengthens weak modality representations through semantic disentanglement and cross-modal enhancement. The framework further incorporates an energy-guided implicit gradient rebalancing mechanism to harmonize inter-modality competition and employs instance-aware modality credibility distillation to adaptively adjust fusion weights. Notably, this approach is the first to unify semantic disentanglement, energy-guided gradient balancing, and sample-level credibility distillation within a single architecture. It achieves state-of-the-art or highly competitive performance under both complete and missing modality settings, significantly enhancing model robustness.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.

Problem

• Multimodal Sentiment Analysis
• Modality Imbalance
• Modality Competition
• Robustness
• Missing Modalities

Innovation

• modality collaboration
• cross-modal enhancement
• gradient rebalancing
• modality trust distillation
• robust multimodal fusion