FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructions

2026-01-08

arXiv.org

AI Summary

This work proposes a zero-shot text-to-speech (TTS) method based on large language models (LLMs) that enables flexible control over speaking style through natural language instructions while accurately cloning speaker timbre using a reference audio. To achieve disentangled control of style and timbre, we introduce a progressive post-training strategy that integrates Direct Preference Optimization (DPO) with a multi-objective Group Relative Policy Optimization (GRPO) algorithm, enabling, for the first time in zero-shot TTS, precise and coordinated manipulation via both natural language prompts and reference speech. Experimental results demonstrate that the proposed approach significantly outperforms existing baselines in style control, voice cloning, and content fidelity. Human evaluations further confirm its superior naturalness, controllability, and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This study proposes FlexiVoice, a text-to-speech (TTS) synthesis system capable of flexible style control with zero-shot voice cloning. The speaking style is controlled by a natural-language instruction and the voice timbre is provided by a speech reference in zero-shot manner. FlexiVoice is built with an LLM core, which takes text as input, and also takes an optional natural language instruction and an optional speech reference to control style and timbre, respectively. FlexiVoice is equipped with a novel Progressive Post-Training (PPT) scheme that progressively unlocks accurate and flexible controllability. In particular, it first employs Direct Preference Optimization (DPO) to enable FlexiVoice to accurately follow both natural language instruction and speech reference simultaneously. It then uses a multi-objective Group Relative Policy Optimization (GRPO) to disentangle style instruction, reference timbre, and textual content. Finally, it adapts instruction GRPO for more advanced instruction following. Experimental results show that FlexiVoice surpasses competing baselines and demonstrates strong capability in decoupling control factors. Human evaluations further confirm its naturalness, controllability, and robustness. Audio samples are available at https://flexi-voice.github.io.

Problem

• zero-shot TTS
• style control
• natural language instruction
• voice cloning
• controllability

Innovation

• Zero-shot TTS
• Natural Language Instruction
• Style Control
• Progressive Post-Training
• Disentangled Voice Synthesis