Every Image Listens, Every Image Dances: Music-Driven Image Animation

2025-01-30

AI Summary

Existing music-driven dance video generation methods suffer from complex modeling, reliance on explicit motion priors, and insufficient beat alignment and text fidelity. This paper introduces the first end-to-end music-and-text dual-conditioned single-image animation framework, eliminating the need for intermediate representations (e.g., pose or depth maps) and enabling zero-shot, personalized animation generation. Our key contributions are: (1) construction of the first large-scale music–text paired dataset comprising 2,904 dance videos; (2) design of a diffusion-based multimodal temporal modeling architecture that jointly encodes audio rhythmic features and textual semantics, augmented with spatiotemporal consistency optimization; and (3) comprehensive outperformance over state-of-the-art baselines in generation quality, beat alignment accuracy, and text-condition fidelity—establishing a new benchmark for music-driven image animation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.

Problem

• Music-driven Dance Generation
• Synchronization
• Video Generation

Innovation

• MuseDance
• diffusion-based technique
• personalized dance video generation