Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition

2025-05-20

AI Summary

Hard parameter sharing in multimodal sentiment analysis (MSA) and multimodal emotion recognition (MER) often induces task interference. To address this, we propose a Multimodal Low-rank Mixture of Experts (MoLE) architecture that decouples shared low-rank experts—capturing cross-task commonalities—from task-specific low-rank experts—modeling modality-task idiosyncrasies. Leveraging low-rank parameterization, our approach effectively mitigates the parameter and computational explosion typically incurred by conventional Mixture-of-Experts (MoE) scaling. The framework jointly encodes textual, acoustic, and visual features to unify MSA and MER within a single model. On CMU-MOSI and CMU-MOSEI benchmarks, it achieves state-of-the-art performance in sentiment analysis and competitive results in emotion recognition. To the best of our knowledge, this is the first work to systematically introduce the low-rank MoE paradigm into multimodal joint sentiment-emotion modeling, offering a novel direction for multitask multimodal learning.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multi-task learning (MTL) enables the efficient transfer of extra knowledge acquired from other tasks. The high correlation between multimodal sentiment analysis (MSA) and multimodal emotion recognition (MER) supports their joint training. However, existing methods primarily employ hard parameter sharing, ignoring parameter conflicts caused by complex task correlations. In this paper, we present a novel MTL method for MSA and MER, termed Multimodal Mixture of Low-Rank Experts (MMoLRE). MMoLRE utilizes shared and task-specific experts to distinctly model common and unique task characteristics, thereby avoiding parameter conflicts. Additionally, inspired by low-rank structures in the Mixture of Experts (MoE) framework, we design low-rank expert networks to reduce parameter and computational overhead as the number of experts increases. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MMoLRE achieves state-of-the-art performance on the MSA task and competitive results on the MER task.

Problem

• Addresses parameter conflicts in multimodal sentiment and emotion analysis
• Proposes low-rank expert networks to reduce computational overhead
• Improves performance on sentiment analysis and emotion recognition tasks

Innovation

• Multimodal Mixture of Low-Rank Experts (MMoLRE)
• Shared and task-specific experts modeling
• Low-rank expert networks reduce overhead