Fast-VGAN: Lightweight Voice Conversion with Explicit Control of F0 and Duration Parameters

2025-07-07

AI Summary

In voice conversion, fine-grained and disentangled control over key acoustic features—such as fundamental frequency (F0), phoneme duration, and phoneme sequence—remains a core challenge. This paper proposes a lightweight end-to-end voice conversion framework that eliminates reliance on traditional vocoders and complex disentanglement modules. Instead, it employs explicit conditional mechanisms to directly govern F0 contours, phoneme durations, and phoneme sequences. Mel-spectrogram generation is modeled using a convolutional network, enhanced with pretrained speaker embeddings, and high-fidelity waveforms are synthesized via a universal neural vocoder. The method achieves high naturalness and audio quality while significantly improving controllability and flexibility. Experiments demonstrate state-of-the-art performance on both speaker conversion and expressive speech synthesis tasks, as evidenced by competitive objective metrics (e.g., MCD, F0 RMSE) and superior subjective evaluations (similarity, naturalness, intelligibility).

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Precise control over speech characteristics, such as pitch, duration, and speech rate, remains a significant challenge in the field of voice conversion. The ability to manipulate parameters like pitch and syllable rate is an important element for effective identity conversion, but can also be used independently for voice transformation, achieving goals that were historically addressed by vocoder-based methods. In this work, we explore a convolutional neural network-based approach that aims to provide means for modifying fundamental frequency (F0), phoneme sequences, intensity, and speaker identity. Rather than relying on disentanglement techniques, our model is explicitly conditioned on these factors to generate mel spectrograms, which are then converted into waveforms using a universal neural vocoder. Accordingly, during inference, F0 contours, phoneme sequences, and speaker embeddings can be freely adjusted, allowing for intuitively controlled voice transformations. We evaluate our approach on speaker conversion and expressive speech tasks using both perceptual and objective metrics. The results suggest that the proposed method offers substantial flexibility, while maintaining high intelligibility and speaker similarity.

Problem

• Precise control of pitch and duration in voice conversion
• Lightweight model for modifying F0 and speaker identity
• Flexible voice transformation without vocoder-based methods

Innovation

• CNN-based model for voice parameter control
• Explicit conditioning on F0 and phonemes
• Universal neural vocoder for waveform conversion