MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis

2025-12-08

AI Summary

Existing domain generalization (DG) methods for multimodal sentiment analysis (MSA) neglect inter-modal collaboration, hindering the modeling of cross-modal semantic invariance; knowledge injection is further constrained by modal fragmentation, leading to representational disintegration. To address these issues, we propose the first DG framework integrating collaborative invariant feature extraction with fine-grained cross-modal knowledge injection. Specifically, we design an Invariant Mixture-of-Experts mechanism to explicitly model inter-modal collaborative invariance; introduce Cross-Modal Adapters to align and fuse knowledge beyond modal boundaries; and jointly employ feature disentanglement and end-to-end optimization to enhance semantic consistency and generalization robustness. Evaluated on three standard benchmarks, our method achieves new state-of-the-art performance, improving average accuracy by 3.2% over prior approaches—demonstrating the efficacy of collaborative modeling and cross-modal knowledge integration for MSA domain generalization.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Existing methods in domain generalization for Multimodal Sentiment Analysis (MSA) often overlook inter-modal synergies during invariant features extraction, which prevents the accurate capture of the rich semantic information within multimodal data. Additionally, while knowledge injection techniques have been explored in MSA, they often suffer from fragmented cross-modal knowledge, overlooking specific representations that exist beyond the confines of unimodal. To address these limitations, we propose a novel MSA framework designed for domain generalization. Firstly, the framework incorporates a Mixture of Invariant Experts model to extract domain-invariant features, thereby enhancing the model's capacity to learn synergistic relationships between modalities. Secondly, we design a Cross-Modal Adapter to augment the semantic richness of multimodal representations through cross-modal knowledge injection. Extensive domain experiments conducted on three datasets demonstrate that the proposed MIDG achieves superior performance.

Problem

• Extracts domain-invariant features to capture inter-modal synergies.
• Enhances multimodal semantic richness via cross-modal knowledge injection.
• Improves domain generalization in multimodal sentiment analysis.

Innovation

• Mixture of Invariant Experts extracts domain-invariant multimodal features
• Cross-Modal Adapter injects knowledge to enrich semantic representations
• Framework enhances inter-modal synergies for domain generalization