SACodec: Asymmetric Quantization with Semantic Anchoring for Low-Bitrate High-Fidelity Neural Speech Codecs

2025-12-23

AI Summary

At ultra-low bitrates, neural speech codecs struggle to simultaneously preserve acoustic fidelity and semantic richness. To address this, we propose a semantic-anchored asymmetric dual-quantization architecture. Our method introduces a novel semantic anchoring mechanism that aligns acoustic features with a frozen mHuBERT codebook via a lightweight projector; designs a residual SimVQ acoustic pathway enabling high-fidelity reconstruction under single-layer quantization; and explicitly decouples semantic and acoustic representation learning. Evaluated at 1.5 kbps, our approach achieves new state-of-the-art performance: subjective MOS scores approach those of the original waveform, while downstream semantic capabilities—e.g., ASR accuracy and emotion recognition—show substantial improvement. To our knowledge, this is the first work to achieve synergistic optimization of both fidelity and semantic expressiveness at such an extreme bitrate.

Technology Category

• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Neural Speech Codecs face a fundamental trade-off at low bitrates: preserving acoustic fidelity often compromises semantic richness. To address this, we introduce SACodec, a novel codec built upon an asymmetric dual-quantizer that employs our proposed Semantic Anchoring mechanism. This design strategically decouples the quantization of Semantic and Acoustic details. The semantic anchoring is achieved via a lightweight projector that aligns acoustic features with a frozen, large-scale mHuBERT codebook, injecting linguistic priors while guaranteeing full codebook utilization. Sequentially, for acoustic details, a residual activation module with SimVQ enables a single-layer quantizer (acoustic path) to faithfully recover fine-grained information. At just 1.5 kbps, SACodec establishes a new state of the art by excelling in both fidelity and semantics: subjective listening tests confirm that its reconstruction quality is perceptually highly comparable to ground-truth audio, while its tokens demonstrate substantially improved semantic richness in downstream tasks.

Problem

• Addresses low-bitrate trade-off between fidelity and semantics
• Decouples quantization of semantic and acoustic details
• Improves semantic richness and fidelity at 1.5 kbps

Innovation

• Asymmetric dual-quantizer decouples semantic and acoustic quantization
• Semantic anchoring aligns features with frozen mHuBERT codebook
• Residual activation module enables fine-grained acoustic detail recovery