MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control

2026-04-22

AI Summary

This work addresses the challenge of achieving fine-grained, token-level control over duration and pauses in high-fidelity text-to-speech (TTS) synthesis. It introduces, for the first time, an explicit token-level modeling framework that conditions on both duration and pause information, integrating high-precision duration supervision, a zero-bias correction mechanism, and a robust training strategy that maintains performance even when control signals are absent. The proposed approach enables natural speech synthesis without explicit timing inputs while supporting targeted, local temporal editing when desired. Experimental results demonstrate that the method significantly outperforms autoregressive baselines on duration control benchmarks and achieves low-bias, reproducible fine-grained timing manipulation in practical applications such as navigation prompts, expressive reading, and accessible code narration.

Technology Category

• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Fine-grained local timing control is still absent from modern text-to-speech systems: existing approaches typically provide only utterance-level duration or global speaking-rate control, while precise token-level timing manipulation remains unavailable. To the best of our knowledge, MAGIC-TTS is the first TTS model with explicit local timing control over token-level content duration and pause. MAGIC-TTS is enabled by explicit token-level duration conditioning, carefully prepared high-confidence duration supervision, and training mechanisms that correct zero-value bias and make the model robust to missing local controls. On our timing-control benchmark, MAGIC-TTS substantially improves token-level duration and pause following over spontaneous synthesis. Even when no timing control is provided, MAGIC-TTS maintains natural high-quality synthesis. We further evaluate practical local editing with a scenario-based benchmark covering navigation guidance, guided reading, and accessibility-oriented code reading. In this setting, MAGIC-TTS realizes a reproducible uniform-timing baseline and then moves the edited regions toward the requested local targets with low mean bias. These results show that explicit fine-grained controllability can be implemented effectively in a high-quality TTS system and can support realistic local timing-editing applications.

Problem

• fine-grained controllable speech synthesis
• local timing control
• token-level duration
• pause control
• text-to-speech

Innovation

• fine-grained controllable TTS
• token-level duration control
• explicit local timing
• pause modeling
• high-confidence duration supervision