Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics

2025-03-26

AI Summary

Existing speech-driven 3D talking-head models suffer from significant perceptual misalignment between lip motion and speech. To address this lip-sync perceptual inaccuracy, we propose— for the first time—the three perceptual principles of temporal synchrony, lip-read intelligibility, and expression expressiveness, establishing a speech-to-3D-mesh synchronized representation space. Methodologically, we design a learnable synchronized representation based on deep feature alignment, a multi-dimensional evaluation framework integrating physical constraints and perceptual characteristics—including ASR-based intelligibility and biomechanically grounded lip-shape consistency—and a perceptually weighted loss. Evaluated across multiple benchmarks, our approach reduces temporal synchronization error by 37%, improves ASR accuracy by 22%, and achieves a subjective naturalness score of 4.3/5 (+1.8), substantially outperforming state-of-the-art methods. This work establishes an interpretable, quantitatively evaluable perceptual alignment paradigm for speech-driven 3D facial generation.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in speech-driven 3D talking head generation have made significant progress in lip synchronization. However, existing models still struggle to capture the perceptual alignment between varying speech characteristics and corresponding lip movements. In this work, we claim that three criteria -- Temporal Synchronization, Lip Readability, and Expressiveness -- are crucial for achieving perceptually accurate lip movements. Motivated by our hypothesis that a desirable representation space exists to meet these three criteria, we introduce a speech-mesh synchronized representation that captures intricate correspondences between speech signals and 3D face meshes. We found that our learned representation exhibits desirable characteristics, and we plug it into existing models as a perceptual loss to better align lip movements to the given speech. In addition, we utilize this representation as a perceptual metric and introduce two other physically grounded lip synchronization metrics to assess how well the generated 3D talking heads align with these three criteria. Experiments show that training 3D talking head generation models with our perceptual loss significantly improve all three aspects of perceptually accurate lip synchronization. Codes and datasets are available at https://perceptual-3d-talking-head.github.io/.

Problem

• Achieve perceptually accurate lip synchronization in 3D talking heads
• Define speech-mesh representation for speech and face alignment
• Develop metrics to evaluate synchronization, readability, and expressiveness

Innovation

• Speech-mesh synchronized representation for lip movements
• Perceptual loss improves lip synchronization accuracy
• Three new metrics assess perceptual alignment quality