MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables

2026-05-28

AI Summary

This work addresses the mismatch between learned representations and downstream objectives in existing speech language models, which typically optimize the encoder and autoregressive model separately, thereby limiting zero-shot text-to-speech (TTS) and speech-to-text (STT) performance. The authors propose a discrete latent variable model based on Mel-spectrograms that enables, for the first time, end-to-end joint training of the encoder and the speech language model. By modeling speech through discrete latent variables, the approach effectively mitigates common autoregressive generation issues such as prolonged silences and word omissions. The method significantly outperforms existing codec- and spectrogram-based baselines in zero-shot TTS and STT tasks, while simultaneously improving representation quality and reducing generation errors.

Abstract

Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel-spectrogram modeling, such as prolonged silence generation and word omissions.

Problem

• speech language modeling
• discrete latent variables
• mel spectrogram
• zero-shot TTS
• encoder optimization

Innovation

• discrete latent variables
• joint optimization
• mel-spectrogram modeling
• speech language modeling
• zero-shot TTS/STT