ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec

2024-06-03

arXiv.org

AI Summary

Existing zero-shot text-to-speech (TTS) models struggle to simultaneously achieve speaker voice cloning and language style control: voice cloning methods lack style controllability, while style-controlled approaches fail to adapt to arbitrary target speakers. ControlSpeech is the first framework to jointly address zero-shot controllability over speaker identity, linguistic content, and prosodic style. It introduces the Style Mixture Semantic Density (SMSD) model, which leverages a disentangled discrete codec space, bidirectional attention, masked parallel decoding, and a Gaussian mixture density network to enable fine-grained style-semantic modeling and diverse, high-fidelity synthesis. Extensive evaluation on a newly curated style-controllable TTS benchmark demonstrates significant improvements in both control accuracy and speech naturalness. To foster reproducibility and community advancement, we open-source ControlToolkit—comprising baseline models, standardized evaluation metrics, and an interactive demo.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging new task-a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture corresponding codec representations in a discrete decoupling codec space. Moreover, we discovered the issue of text style controllability in a many-to-many mapping fashion and proposed the Style Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module which is based on Gaussian mixture density networks, is designed to enhance the fine-grained partitioning and sampling capabilities of style semantic information and generate speech with more diverse styles. In terms of experiments, we make available a controllable model toolkit called ControlToolkit with a new style controllable dataset, some replicated baseline models and propose new metrics to evaluate both the control capability and the quality of generated audio in ControlSpeech. The relevant ablation studies validate the necessity of each component in ControlSpeech is necessary. We hope that ControlSpeech can establish the next foundation paradigm of controllable speech synthesis. The relevant code and demo are available at https://github.com/jishengpeng/ControlSpeech .

Problem

• Simultaneous speaker cloning and style control in TTS
• Decoupling timbre, content, and style in speech synthesis
• Resolving many-to-many issues in textual style control

Innovation

• Bidirectional attention for decoupling codec representations
• Mask-based parallel decoding for simultaneous control
• Gaussian mixture density networks for style resolution