VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

2026-08-13

AI Summary

This study addresses the critical challenges of high latency and the trade-off between audio fidelity and real-time interruption in voice interaction models. We propose a low-latency, continuous streaming text-to-speech (TTS) model designed for intelligent agents, driven directly by LLM text streams. By innovatively incorporating control tokens and a dynamic silence generation mechanism, our approach enables mid-utterance interruptions without KV cache resets and supports silent output during idle states. The system maintains a modular architecture and high-fidelity audio quality while achieving persistent, always-on real-time responsiveness. Consequently, this work effectively resolves the fundamental difficulty in traditional architectures of balancing rapid response times with natural conversational interactivity.

Abstract

Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.

Problem

• Low-Latency Speech Synthesis
• Interactive Agents
• User Barge-in
• Continuous TTS
• Duplex Dialogue

Innovation

• Low-Latency TTS
• Streamable Speech Synthesis
• Mid-Utterance Interruption
• Control Tokens
• KV Cache Preservation