🤖 AI Summary
Existing music-driven virtual human video generation methods struggle to model camera motion and long-term temporal coherence. To address this, we propose the first music-video generation framework with explicit camera control, featuring a multi-stage cascaded architecture: (1) the MV-Director module enables interpretable shot planning and synchronized audio-motion-camera alignment; (2) a temporally aware diffusion Transformer captures long-range spatiotemporal dependencies; and (3) a latent-space camera adapter combined with audio-embedding-guided dynamic denoising enhances motion naturalness. To support training, we introduce Music-in-the-Wild, a large-scale, diverse dataset of in-the-wild musical performances. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches across multiple benchmarks, enabling synthesis of high-fidelity, highly coherent music performance videos lasting several minutes—complete with realistic, controllable camera motion.
📝 Abstract
While diffusion model for audio-driven avatar video generation have achieved notable process in synthesizing long sequences with natural audio-visual synchronization and identity consistency, the generation of music-performance videos with camera motions remains largely unexplored. We present YingVideo-MV, the first cascaded framework for music-driven long-video generation. Our approach integrates audio semantic analysis, an interpretable shot planning module (MV-Director), temporal-aware diffusion Transformer architectures, and long-sequence consistency modeling to enable automatic synthesis of high-quality music performance videos from audio signals. We construct a large-scale Music-in-the-Wild Dataset by collecting web data to support the achievement of diverse, high-quality results. Observing that existing long-video generation methods lack explicit camera motion control, we introduce a camera adapter module that embeds camera poses into latent noise. To enhance continulity between clips during long-sequence inference, we further propose a time-aware dynamic window range strategy that adaptively adjust denoising ranges based on audio embedding. Comprehensive benchmark tests demonstrate that YingVideo-MV achieves outstanding performance in generating coherent and expressive music videos, and enables precise music-motion-camera synchronization. More videos are available in our project page: https://giantailab.github.io/YingVideo-MV/ .