WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

2026-07-07

AI Summary

Current large language model (LLM)-driven text-to-speech (TTS) systems struggle to enable explicit, fine-grained control over word-level acoustic attributes—such as duration, energy, pitch, and intonation—limiting their applicability in domains like audiobook narration and video dubbing. To address this, this work proposes WordVoice, a novel framework that introduces a boundary-token mechanism to facilitate explicit “acoustic planning” within LLMs. Leveraging the newly curated WordVoice-5A dataset comprising 4.7 thousand hours of bilingual speech, and integrating a fine-grained acoustic modulation module, WordVoice achieves, for the first time in LLM-based TTS, disentangled control over multiple word-level acoustic dimensions. The approach supports both adaptive prosody planning and manual intervention, significantly enhancing the precision of independent control across five acoustic attributes while preserving zero-shot synthesis stability and improving overall speech naturalness.

Abstract

While recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal alignment, such as audiobook narration and video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-token mechanism within the LLM to formulate an explicit ``acoustic planning'' process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.

Problem

• word-level control
• LLM-based TTS
• fine-grained prosody
• explicit control
• multi-dimensional acoustic attributes

Innovation

• word-level control
• explicit acoustic planning
• bound-token mechanism
• multi-dimensional prosody
• fine-grained TTS