Affect Decoding in Phonated and Silent Speech Production from Surface EMG

2026-03-12

AI Summary

This study investigates how emotions modulate speech production, particularly the relationship between affective states and articulatory muscle activity during silent speech—an aspect that remains poorly understood. By leveraging facial and neck surface electromyography (sEMG) signals recorded during both vocalized and silent speech tasks, the authors construct a dataset comprising 2,780 samples from 12 participants. They demonstrate for the first time that emotional signatures are embedded in facial motor activity and remain discernible even in the absence of vocal output, offering a novel pathway for affective sensing in silent speech interfaces. Through multimodal feature extraction and model ablation analyses, the proposed approach achieves an AUC of 0.845 in classifying frustration and exhibits strong generalization across vocalized and silent speaking conditions.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The expression of affect is integral to spoken communication, yet, its link to underlying articulatory execution remains unclear. Measures of articulatory muscle activity such as EMG could reveal how speech production is modulated by emotion alongside acoustic speech analyses. We investigate affect decoding from facial and neck surface electromyography (sEMG) during phonated and silent speech production. For this purpose, we introduce a dataset comprising 2,780 utterances from 12 participants across 3 tasks, on which we evaluate both intra- and inter-subject decoding using a range of features and model embeddings. Our results reveal that EMG representations reliably discriminate frustration with up to 0.845 AUC, and generalize well across articulation modes. Our ablation study further demonstrates that affective signatures are embedded in facial motor activity and persist in the absence of phonation, highlighting the potential of EMG sensing for affect-aware silent speech interfaces.

Problem

• affect decoding
• silent speech
• surface EMG
• articulatory execution
• emotion modulation

Innovation

• affect decoding
• surface EMG
• silent speech
• emotion recognition
• articulatory modulation