Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance

2024-09-16

arXiv.org

AI Summary

This study investigates how annotation modality—speech-only versus audiovisual—affects speech emotion recognition (SER) model performance. We systematically evaluate the training efficacy of emotion labels derived from different perceptual modalities within mainstream SER frameworks, employing cross-dataset comparative experiments, controlled-label-source analysis, and multi-condition robustness evaluation. Our key finding is that speech-only annotations significantly enhance model generalization over audiovisual annotations, primarily due to modality alignment between annotation source and model input. Building on this insight, we propose the “fully inclusive label” strategy: integrating multimodal annotation information while using speech modality as the primary supervisory signal. Experiments demonstrate that this strategy achieves state-of-the-art performance on standard benchmarks. The work provides both theoretical grounding and practical guidance for SER annotation paradigms.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Speech
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in different ways. For instance, the IEMOCAP dataset uses video clips with sounds for annotators to provide their emotional perceptions. However, the most significant English emotion dataset, the MSP-PODCAST, only provides speech for raters to choose the emotional ratings. Nevertheless, using speech as input is the standard approach to training SER systems. Therefore, the open question is the emotional labels elicited by which scenarios are the most effective for training SER systems. We comprehensively compare the effectiveness of SER systems trained with labels elicited by different modality stimuli and evaluate the SER systems on various testing conditions. Also, we introduce an all-inclusive label that combines all labels elicited by various modalities. We show that using labels elicited by voice-only stimuli for training yields better performance on the test set, whereas labels elicited by voice-only stimuli.

Problem

• Multisensory Information
• Emotional Speech Recognition
• System Optimization

Innovation

• Sensory Stimulation
• Emotion Recognition
• Integrated Labeling System