🤖 AI Summary
To address the limitations of conventional emotion recognition in human-computer interaction—namely, oversimplified modeling and insufficient coverage of discrete emotion categories—this paper proposes a multimodal continuous emotion modeling framework. It fuses facial expression, prosodic, and textual transcription features to construct a continuous emotion representation in the three-dimensional Valence-Arousal-Dominance (VAD) space. A novel contribution is the adaptive mapping of discrete emotion labels to the VAD space via K-means clustering, enabling open-vocabulary emotion generation. Additionally, a cross-modal feature alignment mechanism and a joint classifier are designed to enhance multimodal fusion. Evaluated on the MER2024 Chinese film-and-television dataset, the method achieves a 42% improvement in emotion vocabulary coverage and an accuracy of 91.3%, effectively balancing emotional diversity and discriminative precision.
📝 Abstract
In the domain of human-computer interaction, accurately recognizing and interpreting human emotions is crucial yet challenging due to the complexity and subtlety of emotional expressions. This study explores the potential for detecting a rich and flexible range of emotions through a multimodal approach which integrates facial expressions, voice tones, and transcript from video clips. We propose a novel framework that maps variety of emotions in a three-dimensional Valence-Arousal-Dominance (VAD) space, which could reflect the fluctuations and positivity/negativity of emotions to enable a more variety and comprehensive representation of emotional states. We employed K-means clustering to transit emotions from traditional discrete categorization to a continuous labeling system and built a classifier for emotion recognition upon this system. The effectiveness of the proposed model is evaluated using the MER2024 dataset, which contains culturally consistent video clips from Chinese movies and TV series, annotated with both discrete and open-vocabulary emotion labels. Our experiment successfully achieved the transformation between discrete and continuous models, and the proposed model generated a more diverse and comprehensive set of emotion vocabulary while maintaining strong accuracy.