Luna-TTS Family Technical Report

2026-08-11

AI Summary

This work addresses key limitations of autoregressive speech synthesis—namely, latency that scales with utterance length, error propagation, and rigid left-to-right generation—by introducing Luna-TTS, a non-autoregressive text-to-speech system based on diffusion language modeling. Pretrained on over one million hours of multilingual speech, Luna-TTS employs a progressive adaptation from causal to block-causal attention to enable both fully parallel and block-autoregressive variants, achieving the first fully non-autoregressive parallel generation over RVQ codebooks. The system natively supports zero-shot voice cloning and inpainting and incorporates reinforcement learning to optimize denoising trajectories. Experimental results show that Luna-TTS achieves state-of-the-art objective metrics on Seed-TTS-Eval and CV3-Eval (CER of 0.73 for Chinese and WER of 1.49 for English), outperforms leading commercial systems in emotional and non-linguistic vocal control, and attains an end-to-end real-time factor as low as 0.0240.

Abstract

Modern text-to-speech (TTS) is dominated by autoregressive (AR) codec language models, whose left-to-right decoding brings latency that grows with utterance length, error accumulation along the committed prefix, and an artificial generation order imposed on the Residual Vector Quantization (RVQ) token grid. We propose Luna-TTS Family, diffusion-language-model-based TTS systems pretrained on 1 million hours of speech across Chinese, English, Japanese, and Korean. The family is built by progressive adaptation of a pretrained AR text LLM, from causal to bidirectional and finally to block-causal attention, and comprises two variants sharing a single tokenizer, data pipeline, and 0.6B backbone lineage. Luna-TTS is fully non-autoregressive: it generates the entire RVQ token grid in a fixed number of parallel refinement steps, with zero-shot voice cloning and speech editing arising natively as infilling. Luna-TTS Realtime, derived by continual training, is autoregressive over blocks of 32 codec frames (1.28s) while denoising each block in parallel; it supports KV-cached blockwise generation and incremental audio delivery, achieving an end-to-end RTF of 0.0240 and 41.6 ms local first-block latency under the warmed serving protocol. An annealed fine-tuning stage adds explicit control over emotion and non-verbal vocalizations (NVVs), and a reinforcement-learning stage applies GRPO with policy ratios computed over the realized denoising trajectory. On Seed-TTS-Eval, Luna-TTS achieves the best results on all four metrics among compared open-source and commercial systems (0.73 CER / 79.7 SIM on test-zh, 1.49 WER / 76.8 SIM on test-en); on the harder in-the-wild CV3-Eval, it posts the lowest Mandarin and English error rates in our comparison. Against leading commercial systems, it achieves the best results on most objective, model-based, and human-rated metrics for NVV and emotion control.

Problem

• autoregressive TTS
• latency
• error accumulation
• RVQ token generation order
• non-autoregressive speech synthesis

Innovation

• non-autoregressive TTS
• diffusion language model
• block-causal attention
• zero-shot voice cloning
• emotion and NVV control