SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset

2026-06-29

AI Summary

This work addresses the limitations of existing speech-driven gesture generation methods, which often overlook cultural differences and are confounded by speaker-specific stylistic variations, thereby hindering accurate modeling of cultural characteristics. To overcome this, we propose SICAGE, a novel framework that learns speaker-invariant cultural representations under a speaker-disjoint data split for the first time. By integrating adversarial learning with Fishr regularization, SICAGE effectively disentangles cultural cues from speaker identity and achieves strong domain generalization. We further introduce TED4C-L, the first large-scale multimodal dataset encompassing four distinct cultural groups, and develop ALaDiT, a real-time diffusion model that efficiently incorporates cultural embeddings. Experiments demonstrate that our approach significantly enhances the generated gestures in terms of naturalness, diversity, beat synchronization, semantic relevance, and cultural consistency.

Abstract

Recent co-speech gesture generation methods often overlook cultural differences, limiting their effectiveness in human-agent interaction. Moreover, culture-conditioned models are rarely evaluated under speaker-disjoint splits, so apparent "cultural" behavior may be confounded with speaker-specific gesturing style. We introduce SICAGE, a modular framework for culture-aware co-speech gesture generation that conditions motion synthesis models on speaker-independent cultural representations. SICAGE learns these representations from audio and text by treating each speaker as a separate domain while imposing invariance across speakers. This encourages representations to remain culture-discriminative while reducing dependence on speaker identity. The resulting cultural embeddings condition a multimodal generator to produce culturally appropriate gestures. We instantiate this idea with two domain generalization approaches: adversarial learning and Fishr regularization. We further introduce ALaDiT, a real-time diffusion-based gesture generator designed to efficiently incorporate the learned cultural embeddings. To validate our method, we built TED4C-L, a 106-hour multimodal dataset of 764 TED speakers from four cultural groups. Experiments show that SICAGE improves motion realism, diversity, beat synchronization, semantic relevance, and cultural consistency.

Problem

• co-speech gesture generation
• cultural differences
• speaker-independent
• human-agent interaction
• cultural representation

Innovation

• culture-aware gesture generation
• speaker-independent representation
• domain generalization
• diffusion-based motion synthesis
• multimodal dataset