DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

2026-07-05

AI Summary

This work addresses the limitations of autoregressive text-to-speech (TTS) models—namely slow inference, error propagation, and hallucination—stemming from their sequential generation of discrete speech tokens. The authors propose a lightweight conversion framework based on LoRA fine-tuning that transforms a pretrained autoregressive TTS model into a discrete diffusion language model, augmented with a confidence-driven non-autoregressive decoding mechanism. Local acoustic context modeling is enhanced via convolutional modules, while a time-shifted sampling schedule and a 1/t-weighted training objective jointly overcome the left-to-right generation constraint. Evaluated on only 585 hours of LibriTTS data, the model achieves a word error rate (WER) of 1.75%, yields a 3.3× speedup in inference, produces more accurate alignments, significantly suppresses hallucinations, and improves decoding confidence.

Abstract

Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.

Problem

• Autoregressive TTS
• diffusion language model
• speech token generation
• inference speed
• hallucination mitigation

Innovation

• diffusion language model
• LoRA adaptation
• confidence-ordered decoding
• convolutional acoustic context
• non-autoregressive TTS