ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion

2026-06-20

AI Summary

Traditional neural speech codecs struggle to disentangle linguistic content, speaker identity, and prosody, often resulting in poor prosody preservation during voice conversion. This work proposes a prosody-oriented codec that models prosody as a conditional residual guided by textual and speaker embeddings, while capturing prosodic variations unexplained by content or speaker through discrete bottleneck representations. By integrating low-frequency Mel-band modeling and training on same-speaker paired data, the method effectively enhances prosody retention. Experimental results demonstrate that the proposed approach significantly improves prosody transfer in voice conversion tasks and substantially reduces source speaker timbre leakage.

Abstract

Neural speech codecs efficiently compress speech and have become a foundation for speech generation, but they are typically learned as holistic representations that intertwine linguistic content, speaker identity, and prosody. While this design is effective for zero-shot voice cloning, it hinders downstream tasks that require prosody preservation or transfer, such as voice conversion. To address this, we introduce ProsoCodec, a prosody-oriented speech codec that models prosody as a conditional residual rather than as a disentangled stream. Specifically, by conditioning both the encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck is encouraged to capture prosodic variation not explained by content and speaker. To further preserve prosody, we use the low-frequency mel band and train the model on paired same-speaker utterances. Experiments on voice conversion show improved prosody preservation and reduced source-timbre leakage.

Problem

• speech codec
• prosody
• voice conversion
• disentanglement
• neural speech representation

Innovation

• Prosody modeling
• Conditional residual
• Speech codec
• Voice conversion
• Disentangled representation