SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting

2025-06-17

AI Summary

To address the lack of visual realism in speech-driven talking-head video synthesis—caused by asynchrony among identity, lip motion, facial expression, and head pose—this paper proposes a high-fidelity, real-time end-to-end framework. Methodologically: (1) we introduce a novel dynamic Gaussian rasterization renderer, integrated with 3D blendshape modeling and a facial synchronization controller, enabling fine-grained, temporally coherent lip and expression generation; (2) we incorporate a head-pose stabilizer and a temporal pose optimization module to ensure natural, smooth head motion; (3) we design an out-of-distribution (OOD)-robust expression generator and a torso refinement module to enhance cross-context generalization. Experiments demonstrate real-time rendering at 101 FPS, with superior performance over state-of-the-art methods in lip-sync accuracy, visual fidelity, and user preference.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic results. To address the critical issue of synchronization, identified as the ''devil'' in creating realistic talking heads, we introduce SyncTalk++, which features a Dynamic Portrait Renderer with Gaussian Splatting to ensure consistent subject identity preservation and a Face-Sync Controller that aligns lip movements with speech while innovatively using a 3D facial blendshape model to reconstruct accurate facial expressions. To ensure natural head movements, we propose a Head-Sync Stabilizer, which optimizes head poses for greater stability. Additionally, SyncTalk++ enhances robustness to out-of-distribution (OOD) audio by incorporating an Expression Generator and a Torso Restorer, which generate speech-matched facial expressions and seamless torso regions. Our approach maintains consistency and continuity in visual details across frames and significantly improves rendering speed and quality, achieving up to 101 frames per second. Extensive experiments and user studies demonstrate that SyncTalk++ outperforms state-of-the-art methods in synchronization and realism. We recommend watching the supplementary video: https://ziqiaopeng.github.io/synctalk++.

Problem

• Achieving high synchronization in speech-driven talking head videos
• Ensuring consistent identity, lip, expression, and pose synchronization
• Enhancing robustness and speed in realistic talking head synthesis

Innovation

• Dynamic Portrait Renderer with Gaussian Splatting
• Face-Sync Controller with 3D blendshape model
• Head-Sync Stabilizer for natural head movements