FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs

2025-09-14

AI Summary

Existing neural audio codecs model only low-level acoustic features, limiting effective integration of semantic and contextual information. Although prior works incorporate self-supervised speech representations or large language model (LLM) embeddings, semantic–contextual alignment and unified learning remain insufficient. This paper proposes FuseCodec, a novel neural codec that jointly models audio, semantic, and contextual representations in the latent space. It achieves strong alignment and end-to-end unified optimization via three key mechanisms: latent-space semantic–contextual fusion, global pooling with broadcast supervision, and temporal dynamic matching. FuseCodec integrates Wav2Vec 2.0-derived semantic representations and LLM-based contextual embeddings without requiring explicit alignment annotations. Evaluated on LibriSpeech, it achieves state-of-the-art performance, significantly outperforming EnCodec, SpeechTokenizer, and DAC across all three core metrics—intelligibility, audio fidelity, and speaker similarity.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Speech tokenization enables discrete representation and facilitates speech language modeling. However, existing neural codecs capture low-level acoustic features, overlooking the semantic and contextual cues inherent to human speech. While recent efforts introduced semantic representations from self-supervised speech models or incorporated contextual representations from pre-trained language models, challenges remain in aligning and unifying the semantic and contextual representations. We introduce FuseCodec, which unifies acoustic, semantic, and contextual representations through strong cross-modal alignment and globally informed supervision. We propose three complementary techniques: (i) Latent Representation Fusion, integrating semantic and contextual features directly into the encoder latent space for robust and unified representation learning; (ii) Global Semantic-Contextual Supervision, supervising discrete tokens with globally pooled and broadcasted representations to enhance temporal consistency and cross-modal alignment; and (iii) Temporally Aligned Contextual Supervision, strengthening alignment by dynamically matching contextual and speech tokens within a local window for fine-grained token-level supervision. We further introduce FuseCodec-TTS, demonstrating our methodology's applicability to zero-shot speech synthesis. Empirically, FuseCodec achieves state-of-the-art performance in LibriSpeech, surpassing EnCodec, SpeechTokenizer, and DAC in transcription accuracy, perceptual quality, intelligibility, and speaker similarity. Results highlight the effectiveness of contextually and semantically guided tokenization for speech tokenization and downstream tasks. Code and pretrained models are available at https://github.com/mubtasimahasan/FuseCodec.

Problem

• Neural codecs overlook semantic and contextual cues in speech
• Challenges in aligning semantic and contextual speech representations
• Need for unified acoustic, semantic and contextual representation learning

Innovation

• Latent Representation Fusion for unified learning
• Global Semantic-Contextual Supervision for alignment
• Temporally Aligned Contextual Supervision for token matching