TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokens

2026-05-29

AI Summary

Current conversational virtual avatars often suffer from rigid facial expressions, audio-visual desynchronization, and fragmented interactivity. This work proposes TokTalk, a novel system that directly leverages streaming audio tokens from Audio-LLMs to drive 3D facial animation in real time, bypassing conventional multi-stage sequential pipelines. To enable this approach, we construct the first dataset mapping audio tokens to 3D facial motion and introduce a lightweight generative model based on chunk-based conditional flow matching, which facilitates low-overhead adaptation to any Audio-LLM. Experiments demonstrate that TokTalk significantly enhances animation quality, expressiveness, and controllability while maintaining low latency, making it well-suited for diverse audio-visual virtual avatar applications.

Abstract

Recent advances in Audio-LLMs like GPT-4o have ushered in an era of conversational interaction with language models. Conversational avatars however, still seem robotic in facial expression and conversational flow, in part due to sequential stages of speech recognition, text generation, turn-based text response, speech synthesis, and audio driven facial animation. Based on our insight that audio-tokens produced by current Audio-LLMs carry sufficient information to reconstruct a plausible facial performance, we present TokTalk, a system that directly outputs expressive facial animation in real-time from streaming audio-tokens. We construct a novel audio-token to 3D facial motion dataset, on which TokTalk is trained using a Chunk-based Conditional Flow Matching model. A lightweight adaptation strategy allows our trained model to seamlessly connect to any token-based Audio-LLM at minimal computational overhead. Our chunk-based processing further enables parametric trade-off between latency and facial quality, shown through ablation studies. We further show that the real-time performance of TokTalk is comparable in latency to prior art solutions, and significantly favorable (via a perceptual study) in terms of quality, expressivity and control of the 3D facial performance. We showcase TokTalk's flexibility using a chatbot Avatar, a voice-driven user Avatar, and an animation Director's interface, as diverse audio-visual face applications.

Problem

• facial animation
• Audio-LLM
• real-time
• expressivity
• conversational avatars

Innovation

• Audio-LLM tokens
• real-time facial animation
• conditional flow matching
• chunk-based processing
• expressive avatars