FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars

2026-06-29

AI Summary

Existing systems struggle to jointly generate speech and facial animations in real-time dialogue, often producing only speech or relying on pre-recorded audio to drive animation. This work proposes FacePlex, the first unified framework enabling full-duplex, online joint generation of speech and facial motion. The core innovations include a formalization of this joint generation task, the introduction of Rolling Flow Matching for streaming facial animation synthesis, and a Rolling Cross-Attention mechanism to model dynamic, mutual conditional dependencies between speech and motion queues. Under strict streaming constraints, experiments demonstrate that FacePlex significantly outperforms existing audio-driven approaches, achieving superior performance in both lip-sync accuracy and facial motion fidelity.

Abstract

Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion. Existing systems only partially address this problem: speech-only full-duplex models can generate speech in real time but do not produce facial motion, while audio-driven facial motion models animate a face from already available audio rather than jointly generating speech and motion online. To bridge this gap, we first formalize full-duplex joint speech-facial motion generation, where speech tokens and facial motion tokens are produced together every step. Building on this formulation, we propose FacePlex, a unified streaming framework with two key components. First, Rolling Flow Matching adapts flow matching to online motion generation by committing new motion frames at each streaming step. Second, Rolling Cross-Attention couples the streaming audio queue with the motion queue, allowing speech and facial motion to condition each other as generation progresses. Through extensive experiments, ablation studies, and a user study, we show that FacePlex enables full-duplex joint speech-facial motion generation under online streaming constraints, while achieving stronger lip-sync quality and motion fidelity than audio-driven facial motion baselines.

Problem

• full-duplex
• joint generation
• speech-facial motion
• conversational avatars
• real-time synchronization

Innovation

• full-duplex generation
• joint speech-facial motion
• Rolling Flow Matching
• Rolling Cross-Attention
• streaming avatar animation