Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

2026-07-11

AI Summary

This work addresses the challenge of training end-to-end models for dance-driven music generation under severe scarcity of paired dance-music data by proposing a framework that effectively leverages both paired and unpaired data. The approach employs pretrained unimodal encoders and aligns cross-modal feature spaces through beat-guided contrastive learning. Furthermore, it integrates a ControlNet-style conditioning mechanism into a text-to-audio diffusion model to enable temporally precise music synthesis aligned with dance motion. Experiments on the AIST++ dataset demonstrate that the proposed method significantly outperforms existing approaches in terms of dance-music alignment while maintaining competitive audio quality.

Abstract

Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .

Problem

• dance-to-music generation
• paired data scarcity
• temporal alignment
• unpaired data

Innovation

• dance-to-music generation
• unpaired data pretraining
• contrastive alignment
• ControlNet-style conditioning
• diffusion model