Autoregressive Speech Synthesis without Vector Quantization

2024-07-11

arXiv.org

AI Summary

Existing autoregressive speech synthesis methods heavily rely on vector quantization (VQ), which disrupts mel-spectrogram continuity and degrades fidelity. This paper introduces MELLE—the first end-to-end autoregressive framework for continuous mel-spectrogram modeling—eliminating VQ to directly generate high-fidelity, continuous-valued mel frames. Its core contributions are: (1) continuous-value token language modeling, overcoming the limitations of discrete tokenization; (2) a spectrogram flux regression loss that explicitly captures temporal dynamics; and (3) a variational inference–driven sampling mechanism that enhances generation diversity and robustness. Evaluated in a single-stage architecture, MELLE surpasses two-stage approaches such as VALL-E across naturalness (MOS), robustness (WER), and fidelity (STOI), establishing a simpler, more efficient paradigm for text-to-speech synthesis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.

Problem

• Autoregressive speech synthesis without vector quantization
• Generates continuous mel-spectrogram frames from text
• Improves fidelity and robustness in text-to-speech

Innovation

• Continuous-valued token modeling for TTS
• Regression loss with spectrogram flux
• Variational inference enhances diversity