Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars

2025-08-22

AI Summary

This work addresses the problem of generating high-fidelity, audio-driven 3D facial animation for digital humans. We propose an end-to-end framework that jointly models speech feature extraction, temporal dynamics, and facial skeletal kinematics, augmented by high-quality multi-subject motion capture data and precise retargeting strategies to enhance cross-subject expression transfer fidelity and consistency. The model enables real-time inference with an end-to-end latency under 80 ms, achieves a lip synchronization error (LSE) of 1.2 mm, and attains high perceptual naturalness, as validated by user studies (MOS 4.6/5.0). To foster reproducibility and deployment, we open-source the training framework, a lightweight SDK, and pre-trained models—providing a scalable, production-ready foundation for interactive digital human systems and game animation pipelines.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Audio-driven facial animation presents an effective solution for animating digital avatars. In this paper, we detail the technical aspects of NVIDIA Audio2Face-3D, including data acquisition, network architecture, retargeting methodology, evaluation metrics, and use cases. Audio2Face-3D system enables real-time interaction between human users and interactive avatars, facilitating facial animation authoring for game characters. To assist digital avatar creators and game developers in generating realistic facial animations, we have open-sourced Audio2Face-3D networks, SDK, training framework, and example dataset.

Problem

• Generating realistic 3D facial animations from audio input
• Enabling real-time interaction between humans and digital avatars
• Providing open-source tools for facial animation authoring in games

Innovation

• Real-time audio-driven facial animation system
• Open-sourced network architecture and training framework
• 3D facial retargeting for digital avatars