Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition

2024-07-31

arXiv.org

AI Summary

Multimodal Emotion Recognition in Conversations (MERC) faces three key challenges: insufficient exploitation of cross-modal cues, multi-source conflicts arising from intra-layer fusion, and difficulty modeling dynamic emotional shifts. To address these, we propose GraphSmile—a novel framework featuring two synergistic modules: Hierarchical Alternating Graph-structured Fusion (GSF) and Explicit Sentiment Dynamics Prediction (SDP). GSF decouples intra- and cross-modal modeling via layered alternating graph neural networks, while SDP explicitly captures temporal sentiment dynamics through an auxiliary sequence modeling task. The method integrates graph neural networks, cross-modal dependency modeling, and multimodal feature disentanglement, enabling unified support for both MERC and Multimodal Sentiment Analysis in Conversations (MSAC). Extensive experiments on multiple benchmark datasets demonstrate significant improvements over state-of-the-art methods, particularly in detecting emotionally abrupt utterances and capturing fine-grained cross-modal affective cues.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. Furthermore, GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), forging a unified multimodal affective model capable of executing MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.

Problem

• Inadequate cross-modal modeling in emotion recognition
• Conflicts from multi-source data fusion
• Difficulty detecting dynamic sentiment changes

Innovation

• Leverages graph structures for inter-modal dependencies
• Alternates intra-modal and inter-modal assimilation layers
• Explicitly models sentiment dynamics between utterances