KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

2026-06-26

AI Summary

Existing audio-driven 3D facial animation methods struggle to simultaneously achieve high fidelity, artistic style control, and frame-level temporal precision, particularly limiting their performance in dialogue localization tasks such as dubbing. This work proposes a keypoint-conditioned flow-based generative framework that decouples audio-driven lip motion from keypoint-driven upper-face dynamics. By incorporating a global style context preservation mechanism and a reference performance-guided temporal control strategy, the method enables high-quality and controllable animation synthesis. Experimental results demonstrate that the proposed approach outperforms state-of-the-art methods in lip-sync accuracy, style consistency, and expressive temporal precision, significantly enhancing both realism and controllability.

Abstract

Speech-driven 3D facial animation methods face significant challenges in simultaneously achieving high-fidelity motion and precise artistic control at production quality. Existing controllable models typically learn global style control by relying on large-scale, low-quality \emph{in-the-wild} datasets that compromise overall animation realism. Furthermore, these frameworks often lack the fine-grained temporal precision required for demanding tasks such as dialogue localization (e.g., dubbing), where matching specific facial expressions is as critical as lip synchronization. We present KM-Speaker (Keypoint-Matching Speaker), a novel keypoint-conditioned flow-based generative framework that provides both global style guidance and frame-level temporal control from reference performances. We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, together with a global style context preservation mechanism to ensure coherent full-face expressiveness. KM-Speaker advances example-based 3D facial animation by achieving high-fidelity motion and flexible controllability in a data-constrained setting, consistently outperforming state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.

Problem

• speech-driven 3D facial animation
• style control
• dialogue localization
• temporal precision
• high-fidelity motion

Innovation

• keypoint-based control
• disentangled facial animation
• flow-based generative model
• dialogue localization
• style-preserving speech-driven animation