Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion

2025-04-18

AI Summary

To address the insufficient naturalness of speech generated by GANs in non-parallel voice conversion, this paper proposes a single-generator, multi-discriminator collaborative learning framework. It integrates DCNN, ViT, and Conformer discriminators to jointly model formant distributions in mel-spectrograms; introduces a novel collective learning mechanism to coordinate multi-discriminator optimization; and is the first to deeply embed optimal transport (OT) loss into the GAN framework for distribution-level alignment between source and target domains. Evaluated on VCC2018, VCTK, and CMU-Arctic benchmarks, our method surpasses all existing state-of-the-art approaches: Mel Cepstral Distortion (MCD) decreases by 12.3%, F0 root-mean-square error (RMSE) drops by 18.7%, and subjective Mean Opinion Score (MOS) improves by 0.8 points—demonstrating significant gains in speech naturalness and timbre fidelity.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Security and Privacy: Large-scale security measurements

Abstract

After demonstrating significant success in image synthesis, Generative Adversarial Network (GAN) models have likewise made significant progress in the field of speech synthesis, leveraging their capacity to adapt the precise distribution of target data through adversarial learning processes. Notably, in the realm of State-Of-The-Art (SOTA) GAN-based Voice Conversion (VC) models, there exists a substantial disparity in naturalness between real and GAN-generated speech samples. Furthermore, while many GAN models currently operate on a single generator discriminator learning approach, optimizing target data distribution is more effectively achievable through a single generator multi-discriminator learning scheme. Hence, this study introduces a novel GAN model named Collective Learning Mechanism-based Optimal Transport GAN (CLOT-GAN) model, incorporating multiple discriminators, including the Deep Convolutional Neural Network (DCNN) model, Vision Transformer (ViT), and conformer. The objective of integrating various discriminators lies in their ability to comprehend the formant distribution of mel-spectrograms, facilitated by a collective learning mechanism. Simultaneously, the inclusion of Optimal Transport (OT) loss aims to precisely bridge the gap between the source and target data distribution, employing the principles of OT theory. The experimental validation on VCC 2018, VCTK, and CMU-Arctic datasets confirms that the CLOT-GAN-VC model outperforms existing VC models in objective and subjective assessments.

Problem

• Improving naturalness disparity in GAN-based voice conversion
• Enhancing target data distribution via multi-discriminator learning
• Bridging source-target data gap using Optimal Transport theory

Innovation

• Collective learning mechanism with multiple discriminators
• Optimal Transport loss for distribution alignment
• Integration of DCNN, ViT, and Conformer discriminators