SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

2026-04-14

AI Summary

This work addresses the challenge of achieving fine-grained, continuous emotional control in speech-driven 3D facial animation, a capability lacking in existing methods that typically rely on utterance-level or manually annotated emotion labels. The authors propose the first frame-level speech emotion segmentation approach to predict both continuous emotion categories and intensities, which are then used as conditioning signals for a novel animation generation model based on a hybrid Transformer-Mamba architecture. By incorporating learnable emotion embeddings, multi-corpus joint training, and an identity-preserving mechanism, the method effectively disentangles linguistic content from emotional style. Evaluated on large-scale emotional datasets, the framework achieves low geometric and temporal reconstruction errors, producing high-quality animations with natural expression transitions and consistent, controllable emotional expressions.

Technology Category

• Natural Language Processing: Speech
• Humans and AI: Emotional Intelligence
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We introduce SEDTalker, an emotion-aware framework for speech-driven 3D facial animation that leverages frame-level speech emotion diarization to achieve fine-grained expressive control. Unlike prior approaches that rely on utterance-level or manually specified emotion labels, our method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. The diarized emotion signals are encoded as learned embeddings and used to condition a speech-driven 3D animation model based on a hybrid Transformer-Mamba architecture. This design allows effective disentanglement of linguistic content and emotional style while preserving identity and temporal coherence. We evaluate our approach on a large-scale multi-corpus dataset for speech emotion diarization and on the EmoVOCA dataset for emotional 3D facial animation. Quantitative results demonstrate strong frame-level emotion recognition performance and low geometric and temporal reconstruction errors, while qualitative results show smooth emotion transitions and consistent expression control. These findings highlight the effectiveness of frame-level emotion diarization for expressive and controllable 3D talking head generation.

Problem

• speech-driven animation
• 3D facial animation
• emotion diarization
• expressive control
• frame-level emotion

Innovation

• frame-level emotion diarization
• emotion-aware animation
• Transformer-Mamba architecture
• expressive 3D facial animation
• speech-driven talking head