Long-Context Speech Synthesis with Context-Aware Memory

2025-08-20

AI Summary

To address the lack of paragraph-level contextual coherence and inconsistent prosody and timbre in long-context text-to-speech (TTS), this paper proposes an end-to-end synthesis framework based on Context-Aware Memory (CAM). The framework dynamically maintains both long-term memory and local context to enable cross-sentence information propagation, and introduces a prefix masking mechanism to support bidirectional contextual modeling under unidirectional autoregressive generation constraints. Its core innovation lies in the synergistic design of the CAM module and a dynamic memory update mechanism, effectively balancing long-range dependency modeling with inference efficiency. Experimental results demonstrate that the proposed method significantly outperforms existing long-context TTS approaches in prosodic expressiveness, paragraph-level coherence, and synthetic naturalness, while maintaining real-time inference speed.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In long-text speech synthesis, current approaches typically convert text to speech at the sentence-level and concatenate the results to form pseudo-paragraph-level speech. These methods overlook the contextual coherence of paragraphs, leading to reduced naturalness and inconsistencies in style and timbre across the long-form speech. To address these issues, we propose a Context-Aware Memory (CAM)-based long-context Text-to-Speech (TTS) model. The CAM block integrates and retrieves both long-term memory and local context details, enabling dynamic memory updates and transfers within long paragraphs to guide sentence-level speech synthesis. Furthermore, the prefix mask enhances the in-context learning ability by enabling bidirectional attention on prefix tokens while maintaining unidirectional generation. Experimental results demonstrate that the proposed method outperforms baseline and state-of-the-art long-context methods in terms of prosody expressiveness, coherence and context inference cost across paragraph-level speech.

Problem

• Addresses paragraph-level coherence loss in long-text speech synthesis
• Resolves style and timbre inconsistencies across extended speech segments
• Reduces context inference cost while improving prosody expressiveness

Innovation

• Context-Aware Memory for dynamic updates
• Prefix mask enables bidirectional attention
• Long-context TTS with memory transfer