MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models

2024-03-14

arXiv.org

AI Summary

To address the urgent need for low-latency, long-sequence (>10 seconds) full-body gesture synthesis in applications such as film, VR, and robotics, existing diffusion models and Transformer-based approaches suffer from high computational overhead and slow inference. This paper proposes a two-stage efficient gesture synthesis framework: first, a VQ-VAE learns a discrete motion prior; second, an enhanced selective state space model (Mamba) performs lightweight multimodal (speech/text)-conditioned temporal modeling. To our knowledge, this is the first work to introduce Mamba into gesture generation, synergistically leveraging discrete motion priors to significantly improve rhythmic accuracy and motion diversity. Experiments demonstrate that our method achieves or surpasses state-of-the-art diffusion and Transformer models across multiple benchmarks, while accelerating inference by 3.2× and enabling real-time, high-fidelity long-sequence generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Manipulation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. Recent advancements have utilized the diffusion model and attention mechanisms to improve gesture synthesis. However, due to the high computational complexity of these techniques, generating long and diverse sequences with low latency remains a challenge. We explore the potential of state space models (SSMs) to address the challenge, implementing a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures. Leveraging the foundational Mamba block, we introduce MambaTalk, enhancing gesture diversity and rhythm through multimodal integration. Extensive experiments demonstrate that our method matches or exceeds the performance of state-of-the-art models.

Problem

• Efficient Gesture Synthesis
• Long Gestures Diversity
• Computational Efficiency

Innovation

• MambaTalk
• State Space Models Optimization
• Gesture Synthesis