Heterogeneous bimodal attention fusion for speech emotion recognition

2025-03-09

AI Summary

To address the heterogeneous modality gap between low-level acoustic features and high-level semantic representations in conversational speech emotion recognition, this paper proposes a multi-level cross-modal interaction framework. The core innovation is the first design of a heterogeneous dual-modal attention fusion mechanism that jointly models absolute and relative cross-modal interactions: context-enhanced unimodal encoding captures modality-specific characteristics; dynamic gated dual-modal attention enables adaptive feature alignment; and cross-modal contrastive learning enforces semantic consistency across modalities at multiple abstraction levels. Evaluated on MELD and IEMOCAP benchmarks, the method achieves new state-of-the-art performance, improving emotion classification accuracy by 2.3–4.1% over prior approaches. Experimental results demonstrate its effectiveness in bridging the representational gap between low-level acoustic signals and high-level linguistic semantics.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multi-modal emotion recognition in conversations is a challenging problem due to the complex and complementary interactions between different modalities. Audio and textual cues are particularly important for understanding emotions from a human perspective. Most existing studies focus on exploring interactions between audio and text modalities at the same representation level. However, a critical issue is often overlooked: the heterogeneous modality gap between low-level audio representations and high-level text representations. To address this problem, we propose a novel framework called Heterogeneous Bimodal Attention Fusion (HBAF) for multi-level multi-modal interaction in conversational emotion recognition. The proposed method comprises three key modules: the uni-modal representation module, the multi-modal fusion module, and the inter-modal contrastive learning module. The uni-modal representation module incorporates contextual content into low-level audio representations to bridge the heterogeneous multi-modal gap, enabling more effective fusion. The multi-modal fusion module uses dynamic bimodal attention and a dynamic gating mechanism to filter incorrect cross-modal relationships and fully exploit both intra-modal and inter-modal interactions. Finally, the inter-modal contrastive learning module captures complex absolute and relative interactions between audio and text modalities. Experiments on the MELD and IEMOCAP datasets demonstrate that the proposed HBAF method outperforms existing state-of-the-art baselines.

Problem

• Bridging heterogeneous modality gap in emotion recognition
• Enhancing multi-modal fusion with dynamic attention mechanisms
• Improving conversational emotion recognition accuracy

Innovation

• Heterogeneous Bimodal Attention Fusion framework
• Dynamic bimodal attention and gating mechanism
• Inter-modal contrastive learning for audio-text interactions