Mitigating Timbre Leakage with Universal Semantic Mapping Residual Block for Voice Conversion

2025-04-11

AI Summary

In voice conversion (VC), speaker timbre often leaks into content representations, degrading target speaker similarity. To address this, we propose a timbre-agnostic content extractor. Our method introduces three key innovations: (1) the first universal phoneme-semantic dictionary built upon multi-speaker statistics; (2) a Content Feature Re-expression (CFR) module with a dual-branch residual architecture, enabling semantic-driven feature disentanglement via phoneme posterior-weighted linear combination; and (3) an explicit residual skip connection mechanism to suppress residual timbre information. The proposed extractor is modular and plug-and-play compatible with various end-to-end VC frameworks. Experiments across multiple benchmarks demonstrate significant suppression of timbre leakage: average improvements of +0.42 in target speaker similarity (SIM) and +0.35 in naturalness (MOS) are achieved.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Voice conversion (VC) transforms source speech into a target voice by preserving the content. However, timbre information from the source speaker is inherently embedded in the content representations, causing significant timbre leakage and reducing similarity to the target speaker. To address this, we introduce a residual block to a content extractor. The residual block consists of two weighted branches: 1) universal semantic dictionary based Content Feature Re-expression (CFR) module, supplying timbre-free content representation. 2) skip connection to the original content layer, providing complementary fine-grained information. In the CFR module, each dictionary entry in the universal semantic dictionary represents a phoneme class, computed statistically using speech from multiple speakers, creating a stable, speaker-independent semantic set. We introduce a CFR method to obtain timbre-free content representations by expressing each content frame as a weighted linear combination of dictionary entries using corresponding phoneme posteriors as weights. Extensive experiments across various VC frameworks demonstrate that our approach effectively mitigates timbre leakage and significantly improves similarity to the target speaker.

Problem

• Reducing timbre leakage in voice conversion systems
• Improving target speaker similarity in converted speech
• Extracting timbre-free content representations from source speech

Innovation

• Residual block with weighted branches
• Universal semantic dictionary for CFR
• Phoneme posteriors as linear combination weights