CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation

2026-09-11

AI Summary

为解决舞蹈到音乐生成中稀疏舞蹈线索与密集音乐信息之间的语义不匹配问题，提出CMA-OT方法，通过外部音乐专家提供层次监督来增强表示学习。

Abstract

Dance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics. Existing methods typically rely on these sparse cues and supervise only the final audio output, resulting in poorly learned music representations and generated music with limited musicality and structural coherence. To address these issues, we propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning. To effectively incorporate hierarchical supervision, we introduce a curriculum-guided multi-scale learning strategy that progressively transfers musical knowledge from the expert to the music generator, enabling stable and effective representation learning. Moreover, to accommodate the semantic and structural variations across different expert scales and achieve fine-grained alignment under temporal mismatch, we propose a scale-aware optimal transport alignment mechanism, which models soft correspondences between hierarchical expert representations and the generator's latent features. Extensive experiments on two datasets demonstrate that CMA-OT achieves state-of-the-art performance in rhythmic synchronization, perceptual quality, and overall music generation.

Problem

• Dance-to-music
• semantic mismatch
• sparse dance cues
• dense information
• music composition

Innovation

• Curriculum-guided Multi-scale representation Alignment
• scale-aware Optimal Transport
• hierarchical supervision
• Dance-to-music generation