DGFM: Full Body Dance Generation Driven by Music Foundation Models

2025-02-27

AI Summary

To address the limited representational capacity of hand-crafted features and the underutilization of music foundation models in music-driven dance generation, this paper proposes a multi-granularity music feature fusion framework. We are the first to incorporate music foundation models—such as MusicLM—into dance generation, jointly leveraging their high-level semantic representations with low-level temporal features (e.g., MFCCs, chroma, tempo) to condition a cross-modal diffusion model. Our method employs hierarchical feature alignment and a gated fusion mechanism to jointly model musical semantics and rhythmic structure, enabling high-fidelity, temporally synchronized 3D full-body dance sequence generation. Extensive evaluations demonstrate significant improvements over four variants of music foundation models and two categories of hand-crafted feature baselines across multiple standard metrics. The generated motions achieve state-of-the-art realism and music-motion temporal alignment accuracy.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

In music-driven dance motion generation, most existing methods use hand-crafted features and neglect that music foundation models have profoundly impacted cross-modal content generation. To bridge this gap, we propose a diffusion-based method that generates dance movements conditioned on text and music. Our approach extracts music features by combining high-level features obtained by music foundation model with hand-crafted features, thereby enhancing the quality of generated dance sequences. This method effectively leverages the advantages of high-level semantic information and low-level temporal details to improve the model's capability in music feature understanding. To show the merits of the proposed method, we compare it with four music foundation models and two sets of hand-crafted music features. The results demonstrate that our method obtains the most realistic dance sequences and achieves the best match with the input music.

Problem

• Generates dance movements using music and text.
• Combines music foundation model with hand-crafted features.
• Enhances dance sequence quality with semantic and temporal details.

Innovation

• Diffusion-based dance generation
• Music foundation models integration
• Enhanced music feature extraction