Adapting a Diffusion-Based Music Synthesis Model to Human Voice Conversion

2026-07-14

AI Summary

Existing diffusion models struggle to unify voice and singing voice conversion with limited generalization capabilities. This work proposes the first adaptation of a multi-instrument music synthesis diffusion model to vocal conversion tasks, leveraging phonetic posteriorgrams (PPGs) and pitch contours as conditioning signals and incorporating feature-wise linear modulation (FiLM) to model speaker/singer identity within a unified speech–singing conversion framework. The approach requires no manual annotations and enables large-scale self-supervised training using off-the-shelf feature extractors. It achieves naturalness and performer similarity on par with or surpassing specialized systems while maintaining precise pitch control, thereby demonstrating the feasibility of cross-domain model transfer. However, it exhibits limitations in phonetic fidelity and experiences audio quality degradation due to the inclusion of instrumental data during training.

Abstract

Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music. Qualitative samples can be found on our project page: https://benadar293.github.io/voice-conversion

Problem

• voice conversion
• diffusion models
• cross-domain adaptation
• speech and singing synthesis
• audio generation

Innovation

• diffusion-based voice conversion
• phonetic posteriorgrams
• feature-wise linear modulation
• cross-domain model transfer
• self-supervised audio representation