POPDG: Popular 3D Dance Generation with PopDanceSet

2024-05-06

Computer Vision and Pattern Recognition

AI Summary

To address weak cross-domain realism, poor rhythmic alignment, and insufficient diversity and physical plausibility in music-driven 3D dance generation, this paper introduces PopDanceSet—the first aesthetics-oriented popular dance dataset tailored to Gen-Z preferences—and proposes POPDG, a novel generative model. Built upon the iD-DPM diffusion framework, POPDG incorporates a spatial enhancement algorithm to enforce biomechanical constraints on joint motion and a lightweight Alignment Module to improve temporal synchronization between music and dance. Evaluated on two benchmark datasets, POPDG achieves state-of-the-art performance, significantly enhancing dance diversity, spatiotemporal coherence, and physical plausibility. Moreover, it pioneers a multi-dimensional evaluation framework for dance generation, extending beyond conventional metrics. Both the PopDanceSet dataset and the POPDG implementation are publicly released.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generating dances that are both lifelike and well-aligned with music continues to be a challenging task in the cross-modal domain. This paper introduces PopDanceSet, the first dataset tailored to the preferences of young audiences, enabling the generation of aesthetically oriented dances. And it surpasses the ${it AIST}++{it dataset}$ in music genre di-versity and the intricacy and depth of dance movements. Moreover, the proposed POPDG model within the iD-DPMframework enhances dance diversity and, through the Space Augmentation Algorithm, strengthens spatial physi-cal connections between human body joints, ensuring that increased diversity does not compromise generation qual-ity. A streamlined Alignment Module is also designed to improve the temporal alignment between dance and mu-sic. Extensive experiments show that POPDG achieves SOTA results on two datasets. Furthermore, the paper also expands on current evaluation metrics. The dataset and code are available at https://github.com/Luke-Luol/POPDG.

Problem

• Cross-domain Synthesis
• Realistic Dance Generation
• Music Synchronization

Innovation

• Dance Generation
• Music Alignment
• 3D Dance Videos