CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents

2026-08-09

AI Summary

Zero-shot speech synthesis faces the challenge of simultaneously achieving high-fidelity audio quality, speaker consistency, and low latency—particularly in streaming systems. This work proposes an efficient, high-quality synthesis framework that integrates a causal variational autoencoder, block-wise autoregressive modeling, explicit speaker conditioning, and a bidirectional flow-matching head. A key innovation is guided-step distillation, which unifies classifier-free guidance with a multi-step solver into a single-step interval-conditioned student model, substantially reducing inference latency while preserving audio fidelity. Experimental results demonstrate competitive intelligibility and speaker similarity on LibriSpeech and Seed-TTS-Eval, with a 23.3% reduction in first-audio latency and a 40.8% improvement in real-time factor.

Abstract

Zero-shot text-to-speech (TTS) now supports interactive assistants, personalized media, and accessibility tools. All TTS systems require faithful linguistic rendering, consistent speaker identity, and low-latency response. Yet compact streaming systems must preserve sufficient acoustic detail in a predictable low-rate latent sequence, while iterative diffusion sampling and classifier-free guidance multiply inference cost at every autoregressive step. To strike a balance between high-fidelity synthesis and low-latency inference, we present CuteTTS, a compact continuous-autoregressive TTS system. It combines semantically aligned causal VAE latents with patch-level autoregression, explicit speaker conditioning, and a bidirectional flow-matching head. We further introduce guidance-step distillation, which absorbs classifier-free guidance and multiple solver steps into a single interval-conditioned student. Evaluations on LibriSpeech and Seed-TTS-Eval demonstrate competitive intelligibility and speaker similarity in zero-shot voice cloning, while distillation lowers first-audio latency by 23.3% and real-time factor by 40.8% relative to the base model with comparable objective and subjective quality. These results provide a practical path toward continuous-autoregressive TTS that reconciles high-fidelity generation with the latency demands of real-time interaction.

Problem

• zero-shot TTS
• low-latency inference
• high-fidelity speech synthesis
• autoregressive modeling
• real-time interaction

Innovation

• continuous autoregressive modeling
• guidance-step distillation
• causal VAE latents
• zero-shot TTS
• flow-matching