On the Effectiveness of ASR Representations in Real-world Noisy Speech Emotion Recognition

2023-11-13

arXiv.org

AI Summary

To address the insufficient robustness of speech emotion recognition (SER) under realistic non-stationary noise, this paper proposes an end-to-end approach that neither requires denoising nor text transcription. Specifically, it leverages intermediate-layer representations from a pre-trained automatic speech recognition (ASR) model as a noise-robust acoustic feature extractor and transfers these representations to emotion classification via feature distillation. This work is the first to directly utilize ASR intermediate representations for non-stationary SER, effectively disentangling non-speech components from noisy signals while avoiding distortions introduced by conventional denoising and biases induced by text dependency. Experiments on realistic noisy scenarios demonstrate that the proposed method significantly outperforms traditional denoising techniques and self-supervised models such as Wav2Vec 2.0, and also surpasses baselines relying on ASR-derived transcriptions or ground-truth text. The approach establishes a novel paradigm for emotion recognition under low-resource and high-noise conditions.

Abstract

This paper proposes an efficient attempt to noisy speech emotion recognition (NSER). Conventional NSER approaches have proven effective in mitigating the impact of artificial noise sources, such as white Gaussian noise, but are limited to non-stationary noises in real-world environments due to their complexity and uncertainty. To overcome this limitation, we introduce a new method for NSER by adopting the automatic speech recognition (ASR) model as a noise-robust feature extractor to eliminate non-vocal information in noisy speech. We first obtain intermediate layer information from the ASR model as a feature representation for emotional speech and then apply this representation for the downstream NSER task. Our experimental results show that 1) the proposed method achieves better NSER performance compared with the conventional noise reduction method, 2) outperforms self-supervised learning approaches, and 3) even outperforms text-based approaches using ASR transcription or the ground truth transcription of noisy speech.

Problem

• Automatic Speech Recognition (ASR)
• Emotion Recognition
• Noisy Environment

Innovation

• Automatic Speech Recognition (ASR)
• Emotion Recognition in Noisy Environments
• Performance Superiority to Traditional Denoising Techniques