🤖 AI Summary
To address the long-standing disconnection between music source separation and multi-track generation tasks, this paper introduces the first unified Multi-track Latent Diffusion Model (MLDM), jointly modeling separation, generation, and orchestration within a single probabilistic framework. By learning an implicit, cross-track-shared musical representation in latent space, MLDM supports unconditional and conditional separation, full-track generation, and arbitrary subset track completion. Trained end-to-end on Slakh2100, MLDM achieves state-of-the-art performance across all core tasks: it improves separation quality by +1.2 dB in SI-SNRi, reduces Fréchet Audio Distance (FAD) by 18% for generation, and significantly enhances orchestration fidelity compared to concurrent models. The model architecture, training code, pretrained weights, and representative audio samples are publicly released.
📝 Abstract
Diffusion models have recently shown strong potential in both music generation and music source separation tasks. Although in early stages, a trend is emerging towards integrating these tasks into a single framework, as both involve generating musically aligned parts and can be seen as facets of the same generative process. In this work, we introduce a latent diffusion-based multi-track generation model capable of both source separation and multi-track music synthesis by learning the joint probability distribution of tracks sharing a musical context. Our model also enables arrangement generation by creating any subset of tracks given the others. We trained our model on the Slakh2100 dataset, compared it with an existing simultaneous generation and separation model, and observed significant improvements across objective metrics for source separation, music, and arrangement generation tasks. Sound examples are available at https://msg-ld.github.io/.