STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs

2026-02-05

AI Summary

Existing neural audio codecs struggle to simultaneously preserve acoustic fidelity and capture semantic content, and incorporating semantic information often degrades reconstruction quality. To address this challenge, this work proposes STACodec, a unified codec framework that introduces a Semantic Token Allocation (STA) mechanism to inject semantic representations from self-supervised models into the first layer of Residual Vector Quantization (RVQ). Furthermore, a Semantic Pre-Distillation (SPD) module is designed to directly predict semantic tokens during inference without relying on external semantic tokenizers. This approach achieves a synergistic optimization of acoustic detail and semantic capability, maintaining high reconstruction fidelity while significantly enhancing performance on downstream semantic tasks.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural audio codecs are widely used for audio compression and can be integrated into token-based language models. Traditional codecs preserve acoustic details well but lack semantic information. Recent hybrid codecs attempt to incorporate semantic information through distillation, but this often degrades reconstruction performance, making it difficult to achieve both. To address this limitation, we introduce STACodec, a unified codec that integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1) via semantic token assignment (STA). To further eliminate reliance on SSL-based semantic tokenizers and improve efficiency during inference, we propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference. Experimental results show that STACodec outperforms existing hybrid codecs in both audio reconstruction and downstream semantic tasks, demonstrating a better balance between acoustic fidelity and semantic capability.

Problem

• audio codecs
• acoustic fidelity
• semantic information
• neural audio compression
• token-based representation

Innovation

• Semantic Token Assignment
• Residual Vector Quantization
• Self-Supervised Learning
• Neural Audio Codec
• Semantic Pre-Distillation