PF-D2M: A Pose-free Diffusion Model for Universal Dance-to-Music Generation

2026-01-22

AI Summary

Existing approaches to dance-to-music generation rely heavily on single human pose features and are constrained by small-scale datasets, limiting their generalization to complex scenarios such as multiple dancers or non-human performers. This work proposes a pose-free diffusion model that directly extracts visual features from dance videos, bypassing the need for explicit pose estimation. To enhance data efficiency and generalization, the method incorporates a progressive training strategy. By operating directly on raw visual inputs, the model accommodates an arbitrary number and type of dancers without requiring pose annotations. Experimental results demonstrate that the proposed approach achieves state-of-the-art performance in both objective metrics and subjective evaluations, setting new benchmarks for dance–music alignment and audio generation quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Dance-to-music generation aims to generate music that is aligned with dance movements. Existing approaches typically rely on body motion features extracted from a single human dancer and limited dance-to-music datasets, which restrict their performance and applicability to real-world scenarios involving multiple dancers and non-human dancers. In this paper, we propose PF-D2M, a universal diffusion-based dance-to-music generation model that incorporates visual features extracted from dance videos. PF-D2M is trained with a progressive training strategy that effectively addresses data scarcity and generalization challenges. Both objective and subjective evaluations show that PF-D2M achieves state-of-the-art performance in dance-music alignment and music quality.

Problem

• dance-to-music generation
• multiple dancers
• non-human dancers
• data scarcity
• generalization

Innovation

• pose-free
• diffusion model
• dance-to-music generation
• visual features
• progressive training