MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

2025-07-28

AI Summary

Existing speech-driven 3D facial animation methods rely on speaker identity labels or pre-defined 3D meshes as priors, limiting their ability to model personalized expression styles and hindering real-world deployment. This paper introduces the first end-to-end audio-to-3D facial animation framework that requires no explicit identity or geometric priors. Our core innovation is a two-stage training paradigm: Stage I learns an audio-driven stylized motion memory mechanism to implicitly encode individual expression dynamics; Stage II jointly optimizes style feature extraction, motion memory storage/retrieval, and 3D dynamic modeling. Quantitative evaluations across multiple benchmarks, visual quality comparisons, and user studies consistently demonstrate that our method significantly outperforms state-of-the-art approaches, achieving breakthroughs in realism, identity consistency, and motion naturalness. This work establishes a new paradigm for low-barrier, high-fidelity personalized digital human generation.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Speech
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: 1-stage is storing and retrieving general motion (i.e., Memorizing), and 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.

Problem

• Generates personalized 3D facial animation from audio
• Eliminates need for speaker labels or 3D meshes
• Learns audio-driven speaking styles for realistic motion

Innovation

• Audio-only input for personalized 3D facial animation
• Two-stage training: memorizing and animating
• Audio-driven style feature for motion emphasis