MEDIC: Zero-shot Music Editing with Disentangled Inversion Control

2024-07-18

arXiv.org

AI Summary

Existing zero-shot audio editing methods (e.g., DDIM inversion) suffer from two key limitations: cumulative diffusion errors and severe content distortion during complex, non-rigid edits—hindering simultaneous preservation of high fidelity and structural integrity. To address this, we propose Decoupled Inversion Control (DIC), a novel framework introducing harmonized attention and a three-branch decoupled inversion strategy—the first to enable collaborative, progressive generation of semantic, harmonic, and melodic components in audio editing. DIC is built upon a diffusion-based disentangled inversion mechanism and accompanied by ZoME-Bench, a new benchmark comprising 1,100 samples across 10 musical categories. Extensive experiments demonstrate that DIC significantly outperforms existing inversion-based methods on zero-shot music editing tasks, achieving substantial improvements in both content fidelity and audio quality. Code, the ZoME-Bench dataset, and audio samples are publicly released.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Text-guided diffusion models revolutionize audio generation by adapting source audio to specific text prompts. However, existing zero-shot audio editing methods such as DDIM inversion accumulate errors across diffusion steps, reducing the effectiveness. Moreover, existing editing methods struggle with conducting complex non-rigid music edits while maintaining content integrity and high fidelity. To address these challenges, we propose MEDIC, a novel zero-shot music editing system based on innovative Disentangled Inversion Control (DIC) technique, which comprises Harmonized Attention Control and Disentangled Inversion. Disentangled Inversion disentangles the diffusion process into triple branches to rectify the deviated path of the source branch caused by DDIM inversion. Harmonized Attention Control unifies the mutual self-attention control and the cross-attention control with an intermediate Harmonic Branch to progressively generate the desired harmonic and melodic information in the target music. We also introduce ZoME-Bench, a comprehensive music editing benchmark with 1,100 samples covering ten distinct editing categories. ZoME-Bench facilitates both zero-shot and instruction-based music editing tasks. Our method outperforms state-of-the-art inversion techniques in editing fidelity and content preservation. The code and benchmark will be released. Audio samples are available at https://medic-edit.github.io/.

Problem

• DDIM inversion accumulates errors reducing editing effectiveness
• Existing methods struggle with complex non-rigid music edits
• Current techniques fail to maintain content integrity and fidelity

Innovation

• Disentangled Inversion Control rectifies DDIM inversion errors
• Harmonized Attention Control unifies self and cross-attention mechanisms
• Triple branch diffusion process preserves content integrity