Language-Codec: Reducing the Gaps Between Discrete Codec Representation and Speech Language Models

2024-02-19

arXiv.org

AI Summary

To address three key bottlenecks in aligning discrete acoustic codecs with speech large language models—training data scale mismatch, multi-codebook redundancy, and first-channel information overload—this paper proposes Mask Channel Residual Vector Quantization (MCRVQ). MCRVQ integrates a channel masking mechanism, an enhanced Fourier-based architecture, and training on 60K hours of speech data, enabling dynamic suppression of information density in the first RVQ channel and streamlined codebook hierarchy. Experiments demonstrate that MCRVQ significantly outperforms state-of-the-art codecs in audio compression. Moreover, in downstream speech-language modeling, it improves acoustic token generation quality and enhances robustness of text-to-speech alignment, achieving co-optimization between codec representations and language model capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In recent years, large language models have achieved significant success in generative tasks (e.g., speech cloning and audio generation) related to speech, audio, music, and other signal domains. A crucial element of these models is the discrete acoustic codecs, which serves as an intermediate representation replacing the mel-spectrogram. However, there exist several gaps between discrete codecs and downstream speech language models. Specifically, 1) most codec models are trained on only 1,000 hours of data, whereas most speech language models are trained on 60,000 hours; 2) Achieving good reconstruction performance requires the utilization of numerous codebooks, which increases the burden on downstream speech language models; 3) The initial channel of the codebooks contains excessive information, making it challenging to directly generate acoustic tokens from weakly supervised signals such as text in downstream tasks. Consequently, leveraging the characteristics of speech language models, we propose Language-Codec. In the Language-Codec, we introduce a Mask Channel Residual Vector Quantization (MCRVQ) mechanism along with improved Fourier transform structures and larger training datasets to address the aforementioned gaps. We compare our method with competing audio compression algorithms and observe significant outperformance across extensive evaluations. Furthermore, we also validate the efficiency of the Language-Codec on downstream speech language models. The source code and pre-trained models can be accessed at https://github.com/jishengpeng/languagecodec .

Problem

• Excessive information in initial codebook channels hinders text-to-acoustic generation
• Multiple codebooks increase downstream speech model complexity
• Bridging gaps between discrete codecs and speech language models

Innovation

• Masked Channel Residual Vector Quantization mechanism
• Improved fourier transform structures
• Refined discriminator design