Problem
Research questions and friction points this paper is trying to address.
Speech-to-Motion
Realism and Diversity
SMPL-X Conversion Limitations
SpeechAct: Towards Generating Whole-body Motion from Speech
🤖 AI Summary
Existing audio-driven full-body motion generation methods suffer from motion distortion, limited diversity, severe foot skating, and difficulty modeling low-audio-correlated body parts (e.g., hands and face). To address these issues, we propose a hybrid point-cloud motion representation integrated with SMPL-X mesh mapping and an encoder-decoder architecture. Our approach introduces a contrastive motion learning mechanism and a robust VQ-VAE to jointly synthesize deterministic facial motions and highly diverse limb motions from audio input. Crucially, it disentangles strongly and weakly audio-correlated motion modules. Quantitative and qualitative evaluations demonstrate significant improvements over state-of-the-art methods in naturalness, motion diversity, and physical plausibility. Our method eliminates foot skating across multiple benchmarks and enables high-fidelity, temporally coherent, and diverse full-body animation—including facial expressions, torso dynamics, and hand gestures.
📝 Abstract
This paper addresses the problem of generating whole-body motion from speech. Despite great successes, prior methods still struggle to produce reasonable and diverse whole-body motions from speech. This is due to their reliance on suboptimal representations and a lack of strategies for generating diverse results. To address these challenges, we present a novel hybrid point representation to achieve accurate and continuous motion generation, e.g., avoiding foot skating, and this representation can be transformed into an easy-to-use representation, i.e., SMPL-X body mesh, for many applications. To generate whole-body motion from speech, for facial motion, closely tied to the audio signal, we introduce an encoder-decoder architecture to achieve deterministic outcomes. However, for the body and hands, which have weaker connections to the audio signal, we aim to generate diverse yet reasonable motions. To boost diversity in motion generation, we propose a contrastive motion learning method to encourage the model to produce more distinctive representations. Specifically, we design a robust VQ-VAE to learn a quantized motion codebook using our hybrid representation. Then, we regress the motion representation from the audio signal by a translation model employing our contrastive motion learning method. Experimental results validate the superior performance and the correctness of our model. The project page is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/SpeechAct.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Speech-driven Motion
Contrastive Learning
VQ-VAE
🔎 Similar Papers
No similar papers found.
