FabasedVC: Enhancing Voice Conversion with Text Modality Fusion and Phoneme-Level SSL Features

2025-11-13

AI Summary

To address the challenge of jointly preserving semantic fidelity and accurately modeling target speaker timbre, prosody, and duration in voice conversion, this paper proposes an end-to-end voice conversion framework built upon VITS. The method innovatively integrates BERT-based text semantic encoding, phoneme-level self-supervised learning (SSL) features, and a learnable duration predictor. Frame-level SSL features are aligned to phonemes via attention-guided phoneme-level average pooling, while explicit phoneme duration constraints enable fine-grained prosody modeling. The duration predictor explicitly controls speech rhythm, enhancing duration consistency. Experiments demonstrate significant improvements over state-of-the-art methods across all three key metrics: naturalness (MOS), speaker similarity (SIM), and content preservation (WER). The proposed approach thus achieves a unified optimization of high-fidelity semantic reconstruction and high-similarity acoustic attribute transfer.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

In voice conversion (VC), it is crucial to preserve complete semantic information while accurately modeling the target speaker's timbre and prosody. This paper proposes FabasedVC to achieve VC with enhanced similarity in timbre, prosody, and duration to the target speaker, as well as improved content integrity. It is an end-to-end VITS-based VC system that integrates relevant textual modality information, phoneme-level self-supervised learning (SSL) features, and a duration predictor. Specifically, we employ a text feature encoder to encode attributes such as text, phonemes, tones and BERT features. We then process the frame-level SSL features into phoneme-level features using two methods: average pooling and attention mechanism based on each phoneme's duration. Moreover, a duration predictor is incorporated to better align the speech rate and prosody of the target speaker. Experimental results demonstrate that our method outperforms competing systems in terms of naturalness, similarity, and content integrity.

Problem

• Preserve semantic information while modeling speaker timbre
• Enhance timbre prosody duration similarity to target speaker
• Improve content integrity through text-phoneme feature integration

Innovation

• Integrates text modality fusion for semantic enhancement
• Uses phoneme-level SSL features with pooling and attention
• Employs duration predictor for target speaker alignment