Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

2026-06-20

AI Summary

This work addresses the limitation of existing full-duplex spoken dialogue systems, which rely solely on audio and neglect facial expressions—a crucial component of human communication. To bridge this gap, we propose Moshi-Face, the first full-duplex dialogue system capable of synchronized audiovisual input and output. Our approach introduces a non-autoregressive framework that jointly generates speech and facial motions with low latency and high temporal alignment. Specifically, we employ a VQ-VAE to construct a facial codec that produces discrete face tokens and integrate a dedicated Face Transformer module into the Moshi architecture. Experimental results demonstrate that Moshi-Face maintains the original audio dialogue quality while enabling high-fidelity, real-time generation of synchronized audio and facial animations.

Abstract

Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.

Problem

• full-duplex spoken dialogue
• facial expressions
• audiovisual alignment
• multimodal communication
• real-time facial generation

Innovation

• full-duplex dialogue
• facial generation
• VQ-VAE
• face tokens
• non-autoregressive generation