PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation

2026-05-08

AI Summary

Existing co-speech gesture generation methods struggle to integrate semantic structure and explicitly disentangle content from style, limiting both semantic consistency and personalized fidelity. This work proposes a two-stage framework: first, a semantics-guided residual vector quantized variational autoencoder (RVQ-VAE) constructs a semantics-aware motion codebook (SMoC), leveraging contrastive learning to achieve explicit content–style disentanglement; second, a masked generative Transformer combined with a cascaded style-residual Transformer synthesizes gesture sequences that are semantically coherent and stylistically controllable, conditioned on reference motion prompts. The method achieves state-of-the-art performance in both objective metrics and user studies, significantly enhancing semantic coherence and style fidelity in generated gestures.

Abstract

Co-speech gesture generation aims to synthesize realistic body movements that are semantically coherent with speech and faithful to a user-specified gestural style. Existing VQ-VAE based co-speech gesture generation methods improve generation quality but fail to encode semantic structure into the motion representation or explicitly disentangle content from style, limiting both semantic coherence and personalization fidelity. We present PersonaGest, a two-stage framework addressing both limitations. In the first stage, a semantic-guided RVQ-VAE disentangles motion content and gestural style within the residual quantization structure, where a Semantic-Aware Motion Codebook (SMoC) organizes the content codebook by gesture semantics and contrastive learning further enforces content-style separation. In the second stage, a Masked Generative Transformer generates content tokens via a semantic-aware re-masking strategy, followed by a cascade of Style Residual Transformers conditioned on a reference motion prompt for style control. Extensive experiments demonstrate state-of-the-art performance on objective metrics and perceptual user studies, with strong style consistency to the reference prompt. Our project page with demo videos is available at https://danny-nus.github.io/PersonaGest/

Problem

• co-speech gesture generation
• semantic coherence
• personalization
• content-style disentanglement
• motion representation

Innovation

• semantic-guided motion representation
• content-style disentanglement
• residual vector quantization
• personalized co-speech gesture generation
• masked generative transformer