MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation

2025-05-31

AI Summary

Existing neural audio codecs prioritize reconstruction fidelity at the expense of semantic interpretability and modeling capability of discrete tokens, limiting their applicability in LLM-driven generative tasks. To address this, we propose MagiCodec: a single-layer streaming Transformer-based audio codec that introduces, for the first time in audio coding, a frequency-domain interpretable Gaussian noise injection mechanism, jointly optimized with latent-space regularization to balance reconstruction accuracy and semantic expressiveness. This design encourages discrete tokens to follow a Zipf distribution, substantially improving compatibility with large language models. Through multi-stage training and discrete quantization, MagiCodec achieves state-of-the-art performance across objective metrics—including LSD and MCD—as well as downstream tasks such as text-to-speech and audio synthesis. It establishes an efficient, semantically grounded representation for joint audio–language modeling.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Neural audio codecs have made significant strides in efficiently mapping raw audio waveforms into discrete token representations, which are foundational for contemporary audio generative models. However, most existing codecs are optimized primarily for reconstruction quality, often at the expense of the downstream modelability of the encoded tokens. Motivated by the need to overcome this bottleneck, we introduce $ extbf{MagiCodec}$, a novel single-layer, streaming Transformer-based audio codec. MagiCodec is designed with a multistage training pipeline that incorporates Gaussian noise injection and latent regularization, explicitly targeting the enhancement of semantic expressiveness in the generated codes while preserving high reconstruction fidelity. We analytically derive the effect of noise injection in the frequency domain, demonstrating its efficacy in attenuating high-frequency components and fostering robust tokenization. Extensive experimental evaluations show that MagiCodec surpasses state-of-the-art codecs in both reconstruction quality and downstream tasks. Notably, the tokens produced by MagiCodec exhibit Zipf-like distributions, as observed in natural languages, thereby improving compatibility with language-model-based generative architectures. The code and pre-trained models are available at https://github.com/Ereboas/MagiCodec.

Problem

• Enhancing semantic expressiveness in audio codecs
• Balancing reconstruction quality and token modelability
• Improving compatibility with language-model architectures

Innovation

• Single-layer streaming Transformer-based audio codec
• Gaussian noise injection for robust tokenization
• Multistage training with latent regularization