M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations

2025-08-26

AI Summary

The Multimodal Emotional Cause Triplet Extraction from Conversations (MECTEC) task faces two key challenges: severe data scarcity—only a single-scene public dataset exists—and insufficient modeling—prior approaches fail to explicitly integrate emotional/causal context and hierarchical multimodal semantics. Method: We introduce MECAD, the first benchmark dataset for MECTEC covering diverse real-world conversational scenarios and modalities, and propose M3HG, a Multimodal Heterogeneous Graph Neural Network. M3HG constructs a dual-level heterogeneous graph (inter- and intra-utterance) and jointly models emotional, causal, and cross-modal semantics via multi-scale node representations and semantically typed edges. Contribution/Results: Extensive experiments on MECAD and existing benchmarks demonstrate that M3HG significantly outperforms state-of-the-art methods, validating its effectiveness and strong generalization capability in complex, multimodal conversational settings.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Emotion Cause Triplet Extraction in Multimodal Conversations (MECTEC) has recently gained significant attention in social media analysis, aiming to extract emotion utterances, cause utterances, and emotion categories simultaneously. However, the scarcity of related datasets, with only one published dataset featuring highly uniform dialogue scenarios, hinders model development in this field. To address this, we introduce MECAD, the first multimodal, multi-scenario MECTEC dataset, comprising 989 conversations from 56 TV series spanning a wide range of dialogue contexts. In addition, existing MECTEC methods fail to explicitly model emotional and causal contexts and neglect the fusion of semantic information at different levels, leading to performance degradation. In this paper, we propose M3HG, a novel model that explicitly captures emotional and causal contexts and effectively fuses contextual information at both inter- and intra-utterance levels via a multimodal heterogeneous graph. Extensive experiments demonstrate the effectiveness of M3HG compared with existing state-of-the-art methods. The codes and dataset are available at https://github.com/redifinition/M3HG.

Problem

• Extracting emotion-cause triplets from multimodal conversational data
• Addressing dataset scarcity in multimodal emotion cause extraction
• Modeling emotional contexts and fusing multi-level semantic information

Innovation

• Multimodal heterogeneous graph captures emotional contexts
• Fuses contextual information at inter-intra utterance levels
• Introduces first multimodal multi-scenario MECTEC dataset