Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

2026-04-26

AI Summary

Existing audio-driven facial animation methods struggle to achieve high-fidelity personalization in real-time streaming scenarios, often relying on audio look-ahead or static embeddings that fail to capture dynamic individual characteristics. This work proposes an end-to-end causal framework that introduces a temporally hierarchical motion representation to jointly model global context and high-frequency details. It further incorporates a multimodal style retrieval mechanism that dynamically fuses audio and motion cues from an arbitrary number of unstructured reference clips, extracting personalized style priors while preserving strict causality. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceptual realism, as rigorously validated through both quantitative metrics and user studies.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Audio-driven facial animation is essential for immersive digital interaction, yet existing frameworks fail to reconcile real-time streaming with high-fidelity personalization. Current methods often rely on latency-inducing audio look-ahead, or require high user compliance to pre-encode static embeddings that fails to capture dynamic idiosyncrasies. We present an end-to-end causal framework for personalizing causal facial motion generation via dynamic multi-modal style retrieval, enabling ultra-low latency while uniquely leveraging unstructured style references. We introduce two key innovations: (1) a temporal hierarchical motion representation that captures global temporal context and high-frequency details while maintaining decoding causality, and (2) a multi-modal style retriever that jointly queries audio and motion to dynamically extract stylistic priors without breaking causality. This mechanism allows for scalable personalization with total flexibility regarding the number and contents of templates. By integrating these components into a causal autoregressive architecture, our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism, supported by extensive quantitative evaluations and user studies.

Problem

• audio-driven facial animation
• personalization
• causality
• low latency
• dynamic idiosyncrasies

Innovation

• causal animation
• dynamic multi-modal retrieval
• temporal hierarchical representation
• audio-driven facial motion
• personalization