Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

2026-03-06

AI Summary

This work addresses the challenges of unnatural prosody and context-unboundedness-induced failures in streaming text-to-speech (TTS) systems, which stem from the lack of lookahead information. The authors propose a prosody-boundary-aware post-training strategy that fine-tunes a pre-trained LLM-TTS model using weakly time-aligned data, enabling it to anticipate content boundaries and stop appropriately with only limited future textual context. During inference, a sliding-window prompting mechanism conveys historical text and speech tokens to maintain bounded context and ensure seamless concatenation. This approach is the first to integrate prosody boundary awareness into streaming TTS post-training, achieving natural pauses without requiring complete sentences. Experiments demonstrate an absolute 66.2% reduction in word error rate (from 71.0% to 4.8%), along with relative improvements of 16.1% and 1.5% in speaker and emotional similarity, respectively, substantially outperforming the CosyVoice-Style baseline.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Streaming TTS that receives streaming text is essential for interactive systems, yet this scheme faces two major challenges: unnatural prosody due to missing lookahead and long-form collapse due to unbounded context. We propose a prosodic-boundary-aware post-training strategy, adapting a pretrained LLM-based TTS model using weakly time-aligned data. Specifically, the model is adapted to learn early stopping at specified content boundaries when provided with limited future text. During inference, a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation. Evaluations show our method outperforms CosyVoice-Style interleaved baseline in both short and long-form scenarios. In long-text synthesis, especially, it achieves a 66.2% absolute reduction in word error rate (from 71.0% to 4.8%) and increases speaker and emotion similarity by 16.1% and 1.5% relatively, offering a robust solution for streaming TTS with incremental text.

Problem

• Streaming TTS
• prosody
• long-form collapse
• lookahead
• unbounded context

Innovation

• prosodic boundary-aware
• streaming TTS
• LLM-based TTS
• sliding-window prompting
• weakly time-aligned data