Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation

2025-07-28

AI Summary

Traditional speech-driven 3D facial animation methods model frames independently, failing to capture coarticulation effects—leading to lip jitter and motion discontinuities. To address this, we propose a phoneme-context-aware viseme modeling paradigm: (1) introducing learnable coarticulation weights that dynamically modulate the influence of surrounding phonemes on facial motion; and (2) designing a phoneme-context-aware loss function that jointly optimizes speech recognition front-end features and 3D facial keypoint regression in an end-to-end manner. Our approach explicitly encodes speech continuity constraints on facial expression transitions. Evaluated on multiple benchmark datasets, it achieves significant improvements in animation smoothness and naturalness—outperforming conventional reconstruction-based methods both quantitatively (e.g., lower jitter metrics, higher landmark consistency) and subjectively (via user studies). The proposed framework effectively mitigates visual jitter while preserving phonetic accuracy and temporal coherence.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Speech-driven 3D facial animation aims to generate realistic facial movements synchronized with audio. Traditional methods primarily minimize reconstruction loss by aligning each frame with ground-truth. However, this frame-wise approach often fails to capture the continuity of facial motion, leading to jittery and unnatural outputs due to coarticulation. To address this, we propose a novel phonetic context-aware loss, which explicitly models the influence of phonetic context on viseme transitions. By incorporating a viseme coarticulation weight, we assign adaptive importance to facial movements based on their dynamic changes over time, ensuring smoother and perceptually consistent animations. Extensive experiments demonstrate that replacing the conventional reconstruction loss with ours improves both quantitative metrics and visual quality. It highlights the importance of explicitly modeling phonetic context-dependent visemes in synthesizing natural speech-driven 3D facial animation. Project page: https://cau-irislab.github.io/interspeech25/

Problem

• Enhancing speech-driven 3D facial animation realism
• Addressing coarticulation-induced jitter in facial motion
• Modeling phonetic context for smoother viseme transitions

Innovation

• Phonetic context-aware loss modeling
• Viseme coarticulation weight adaptation
• Enhanced smooth facial animation synthesis