OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

2026-06-29

AI Summary

This work addresses the limitations of existing dance video generation methods, which suffer from a scarcity of high-quality data and insufficient integration of music with generative models. To overcome these challenges, the authors introduce CIPE-Dance, a large-scale dataset comprising 300,000 high-fidelity dance video clips, and propose OmniDance—a unified framework capable of high-fidelity dance synthesis driven by text (TI2V), music (MI2V), or multimodal inputs (MTI2V). OmniDance innovatively integrates a depth-aware architecture, anchor-based curriculum learning, and a modality-specific time-varying classifier-free guidance (CFG) mechanism. Rigorous expert-guided data curation and annotation further enhance dataset quality. Evaluated on CIPE-Dance, OmniDance achieves state-of-the-art performance across all three generation tasks, significantly improving motion-music rhythmic alignment and visual fidelity.

Abstract

Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models. To address these limitations, we introduce CIPE-Dance, a large-scale Internet-sourced dance video dataset with choreography-informed text annotations, constructed via a progressive expert pipeline. To the best of our knowledge, CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours and covering diverse dancers, environments, and dance genres. We further propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing its original controllability or visual fidelity. Motivated by the complementary roles of text as low-frequency semantics and music as high-frequency temporal dynamics, OmniDance co-designs a depth-aware specialization architecture, an anchored easy-to-hard curriculum learning strategy, and a modality-specialized time-dependent CFG strategy, enabling unified TI2V, MI2V, and MTI2V generation. Extensive experiments on CIPE-Dance demonstrate that OmniDance achieves state-of-the-art performance across all three tasks and exhibits robust multimodal integration capability. Project is available at https://github.com/AMAP-ML/OmniDance.

Problem

• dance video generation
• music-driven generation
• multimodal integration
• video foundation models
• large-scale dataset

Innovation

• dance video generation
• multimodal integration
• foundation model
• music-conditioned generation
• large-scale dataset