AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation

2026-06-20

AI Summary

This work addresses the challenge of achieving high-fidelity speech reconstruction and effective disentanglement of semantic, speaker, and prosodic attributes at an extremely low bitrate of 12.5 Hz. The authors propose a three-stream token modeling approach based on tailored data augmentation, which decomposes speech into three independent representations corresponding to semantics, speaker identity, and prosody. To further enhance semantic encoding, they introduce an augmented alignment loss that optimizes the semantic encoder’s output. Experimental results demonstrate that the proposed method significantly improves both reconstruction fidelity and attribute disentanglement under low-bitrate conditions, outperforming state-of-the-art approaches on the LibriSpeech test-clean benchmark and validating the efficacy of the designed disentanglement mechanism and loss function.

Abstract

We propose AugCodec, a low-bitrate disentangled neural speech codec that leverages data augmentation to decompose speech into three distinct components: semantic, speaker, and prosody tokens. Specifically, we employ tailored augmenta tion strategies to transform speech into distinct variants, each serving as input for extracting tokens that preserve the target attribute while suppressing others. This disentanglement strategy enables substantial reduction in token rate. Further more, we introduce an augmentation loss that aligns semantic encoder outputs between source and voice-converted speech, encouraging speaker-agnostic embeddings while mitigating the acoustic mismatch induced by voice conversion. Experiments on LibriSpeech test-clean demonstrate that AugCodec significantly outperforms state-of-the-art methods in both reconstruction quality and disentanglement, while operating at only 12.5Hz with three token streams.

Problem

• low-bitrate
• disentangled speech representation
• neural speech codec
• speech decomposition
• data augmentation

Innovation

• disentangled speech representation
• data augmentation
• low-bitrate neural codec
• semantic-speaker-prosody decomposition
• augmentation loss