MAJIC: Leveraging Articulatory Motion for Speech-based Emotion Recognition

2026-06-16

AI Summary

This study addresses the limited performance of existing speech emotion recognition systems in capturing natural and subtle emotional expressions. To overcome this challenge, the work proposes a multimodal approach that systematically integrates articulatory motion features—such as jaw displacement and facial muscle tremors—as a complementary modality to audio signals for the first time. These modalities are effectively fused within a multi-task learning framework. Evaluated on a diverse dataset comprising 20 speakers, 10 languages, and varied conversational scenarios, the proposed method achieves 93% accuracy and a 91% F1 score, substantially outperforming strong audio-only baselines. These results demonstrate that articulatory motion provides critical cues for enhancing fine-grained emotion recognition capabilities.

Abstract

We introduce MAJIC, a multimodal emotion recognition system that leverages articulatory motion of the jaw and facial muscles for speech-based emotion recognition (SER). While most SER systems perform well on datasets with strongly expressed emotional speech of trained actors, their performance often degrades when emotional expressions become more subtle. We explore this challenge by engineering features from articulatory motion and integrating them with audio features using a multi-task learning framework. Our key insight is that emotion in speech manifests not only through vocal characteristics but also through distinct articulatory motions: jaw movements, facial muscle vibrations, and speech-induced vibrations. While audio captures features such as pitch and prosody, articulatory motion contains complementary information that is not present in audio alone. We evaluate our system on data collected from 20 participants across multiple sessions, 10 languages, and diverse scenarios, including prompted and conversational speech, showing its robustness across users and settings. MAJIC achieves 93% accuracy and 91% F1 score for emotion classification, outperforming strong audio-based baselines on our dataset.

Problem

• speech-based emotion recognition
• subtle emotional expressions
• articulatory motion
• multimodal emotion recognition
• emotion classification

Innovation

• articulatory motion
• multimodal emotion recognition
• speech-based emotion recognition
• multi-task learning
• facial muscle vibration