UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction

2026-01-06

arXiv.org

AI Summary

This work proposes UniSRCodec, a single-codebook neural audio codec that achieves high-fidelity reconstruction at high sampling rates and an ultra-low bitrate of only 40 tokens per second. Existing single-codebook approaches struggle with unified modeling, high-fidelity synthesis, and accurate high-frequency reconstruction, while multi-codebook methods often suffer from architectural complexity and poor compatibility with downstream tasks. UniSRCodec addresses these limitations through a novel integration of a Mel-spectrogram-based time-frequency compression mechanism and a subband reconstruction module, enabling high-quality cross-band audio reconstruction within a single-codebook framework. Phase information is recovered via a dedicated vocoder. Experimental results demonstrate that UniSRCodec attains state-of-the-art performance among single-codebook codecs in cross-domain audio reconstruction, delivering audio quality comparable to certain multi-codebook systems while maintaining structural simplicity and strong downstream task compatibility.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Neural Audio Codecs (NACs) can reduce transmission overhead by performing compact compression and reconstruction, which also aim to bridge the gap between continuous and discrete signals. Existing NACs can be divided into two categories: multi-codebook and single-codebook codecs. Multi-codebook codecs face challenges such as structural complexity and difficulty in adapting to downstream tasks, while single-codebook codecs, though structurally simpler, suffer from low-fidelity, ineffective modeling of unified audio, and an inability to support modeling of high-frequency audio. We propose the UniSRCodec, a single-codebook codec capable of supporting high sampling rate, low-bandwidth, high fidelity, and unified. We analyze the inefficiency of waveform-based compression and introduce the time and frequency compression method using the Mel-spectrogram, and cooperate with a Vocoder to recover the phase information of the original audio. Moreover, we propose a sub-band reconstruction technique to achieve high-quality compression across both low and high frequency bands. Subjective and objective experimental results demonstrate that UniSRCodec achieves state-of-the-art (SOTA) performance among cross-domain single-codebook codecs with only a token rate of 40, and its reconstruction quality is comparable to that of certain multi-codebook methods. Our demo page is available at https://wxzyd123.github.io/unisrcodec.

Problem

• Neural Audio Codecs
• single-codebook codec
• high-frequency audio
• low-fidelity
• unified audio modeling

Innovation

• single-codebook codec
• sub-band reconstruction
• neural audio codec
• low-bitrate compression
• Mel-spectrogram