AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

2026-08-17

AI Summary

This study addresses the reliance on character-specific data and labor-intensive rigging in arbitrary 3D speech animation generation by proposing a zero-animation-data-driven paradigm. The method leverages a video diffusion model to convert talking-head videos into 3D animations, employing character-specific fine-tuning to preserve motion priors. Furthermore, it integrates blendshape parameter optimization with knowledge distillation to construct a real-time inference network. This approach achieves cross-mesh, zero-shot 3D lip-sync generation without requiring any 3D training data. By eliminating the need for 3D datasets, the proposed framework significantly reduces manual annotation costs while maintaining real-time performance, thereby offering an efficient solution for universal 3D speech animation.

Abstract

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.

Problem

• 3D speech animation
• arbitrary characters
• audio-driven
• animation data
• video generation model

Innovation

• Video Diffusion Model
• Character-specific Fine-tuning
• Zero-shot Animation
• Blendshape Optimization
• Real-time Distillation