CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio

2025-09-11

AI Summary

Existing audio autoencoders struggle to simultaneously achieve high compression ratios and high-fidelity reconstruction, while continuous embeddings and discrete token representations often involve trade-offs. This paper proposes CoDiCodec—the first unified audio autoencoder capable of **jointly outputting both continuous embeddings and discrete tokens**. Its core innovation is FSQ-dropout, an end-to-end trainable mechanism requiring no auxiliary loss; combined with finite scalar quantization (FSQ), consistency regularization, and parallel decoding, it achieves efficient compression at ~11 Hz continuous sampling rate and 2.38 kbps discrete bitrate. Experiments demonstrate that CoDiCodec significantly outperforms state-of-the-art continuous and discrete models in reconstruction quality at comparable bitrates. Moreover, it supports multitask generative applications while delivering high compression efficiency, low latency, and superior perceptual fidelity.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Sustainability of Web economics

Abstract

Efficiently representing audio signals in a compressed latent space is critical for latent generative modelling. However, existing autoencoders often force a choice between continuous embeddings and discrete tokens. Furthermore, achieving high compression ratios while maintaining audio fidelity remains a challenge. We introduce CoDiCodec, a novel audio autoencoder that overcomes these limitations by both efficiently encoding global features via summary embeddings, and by producing both compressed continuous embeddings at ~ 11 Hz and discrete tokens at a rate of 2.38 kbps from the same trained model, offering unprecedented flexibility for different downstream generative tasks. This is achieved through Finite Scalar Quantization (FSQ) and a novel FSQ-dropout technique, and does not require additional loss terms beyond the single consistency loss used for end-to-end training. CoDiCodec supports both autoregressive decoding and a novel parallel decoding strategy, with the latter achieving superior audio quality and faster decoding. CoDiCodec outperforms existing continuous and discrete autoencoders at similar bitrates in terms of reconstruction audio quality. Our work enables a unified approach to audio compression, bridging the gap between continuous and discrete generative modelling paradigms.

Problem

• Unifying continuous and discrete audio compressed representations
• Achieving high compression ratios with audio fidelity
• Overcoming limitations of existing autoencoders for generative tasks

Innovation

• Unifies continuous and discrete audio representations
• Uses Finite Scalar Quantization with FSQ-dropout technique
• Supports both autoregressive and parallel decoding strategies