SDP-Codec: A Speaker-Decoupled Speech Codec with Pitch Injection for Low-Bitrate Coding and Zero-Shot Voice Conversion

2026-06-19

AI Summary

This work proposes a single-stage disentangled neural speech codec that simultaneously achieves high-fidelity reconstruction and effective suppression of speaker information at low bitrates, overcoming limitations of existing approaches that often rely on multi-stage training or retain residual speaker characteristics. The proposed framework uniquely integrates normalized fundamental frequency (F0) injection, global conditional de-normalization, and a soft-label pitch reconstruction objective within a unified architecture, while leveraging continuous pre-quantized features from a pretrained self-supervised encoder to capture local content representations. Evaluated at both 16 kHz and 24 kHz sampling rates, the model demonstrates superior reconstruction quality and zero-shot voice conversion performance, alongside the lowest speaker identification accuracy at comparable bitrates—confirming its enhanced disentanglement efficacy.

Abstract

Speaker-decoupled speech codecs can reduce bitrate by separating global speaker attributes from local content and prosody, while supporting voice conversion. Existing speaker-decoupled codecs face a trade-off: methods that explicitly suppress speaker leakage often rely on multi-stage or auxiliary training, whereas simpler designs can leave residual speaker information in local tokens. We propose SDP-Codec, a speaker-decoupled, pitch-injected codec trained with a single-stage optimization pipeline. SDP-Codec derives local tokens from continuous pre-quantization features of a pretrained self-supervised encoder and injects normalized F0 via a pitch encoder-decoder with global-conditioned denormalization and soft-label pitch reconstruction objective. Across 16 kHz and 24 kHz settings, SDP-Codec achieves competitive reconstruction and strong zero-shot voice conversion at comparable bitrates, with the lowest speaker-probing accuracy among compared systems, suggesting reduced speaker leakage.

Problem

• speaker-decoupled
• speech codec
• speaker leakage
• low-bitrate coding
• zero-shot voice conversion

Innovation

• speaker-decoupled codec
• pitch injection
• zero-shot voice conversion
• single-stage training
• speaker leakage reduction