Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer

2024-09-09

arXiv.org

AI Summary

This work addresses music timbre transfer under unpaired data conditions—transforming source-instrument audio into target-instrument timbre while preserving melodic structure. We propose the Dual Diffusion Bridge (DDB), a novel architecture comprising two sequential diffusion processes: a forward diffusion bridge that progressively maps the source audio’s latent representation to a Gaussian prior, followed by a reverse diffusion process conditioned on the target instrument to reconstruct the timbre-transferred audio. Theoretically and empirically, we show that the noise scale σ continuously governs the trade-off between timbre transfer fidelity and melodic preservation. Evaluated on the CocoChorales dataset, our method achieves significant improvements over VAEGAN and Gaussian Flow Bridges in both Fréchet Audio Distance (FAD) and Diatonic Pitch Distance (DPD), demonstrating superior timbre conversion quality and strong melodic consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Music timbre transfer is a challenging task that involves modifying the timbral characteristics of an audio signal while preserving its melodic structure. In this paper, we propose a novel method based on dual diffusion bridges, trained using the CocoChorales Dataset, which consists of unpaired monophonic single-instrument audio data. Each diffusion model is trained on a specific instrument with a Gaussian prior. During inference, a model is designated as the source model to map the input audio to its corresponding Gaussian prior, and another model is designated as the target model to reconstruct the target audio from this Gaussian prior, thereby facilitating timbre transfer. We compare our approach against existing unsupervised timbre transfer models such as VAEGAN and Gaussian Flow Bridges (GFB). Experimental results demonstrate that our method achieves both better Fr'echet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB. Additionally, we discover that the noise level from the Gaussian prior, $sigma$, can be adjusted to control the degree of melody preservation and amount of timbre transferred.

Problem

• Automatic Music Timbre Conversion
• Melody Preservation
• Computer Music Processing

Innovation

• Dual Diffusion Bridge
• Melody Preservation
• CocoChorales Dataset