VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

2026-06-24

AI Summary

This work addresses the limited effectiveness of zero-shot text-to-speech (TTS) systems in mimicking rare speaking styles such as cross-talk (xiangsheng) and regional dialects, where conventional fine-tuning approaches require substantial high-quality data and hinder rapid personalization. To overcome this, the authors propose VoiceTTA, the first framework integrating reinforcement learning with test-time adaptation to enhance style imitation by optimizing learnable prefixes of a pretrained zero-shot TTS model. The approach constructs a style-aware reward based on differences in fundamental frequency and energy coefficient of variation, complemented by speaker similarity and intelligibility signals. Leveraging a flow-matching architecture and Whisper-based word error rate estimation, VoiceTTA employs Group Relative Preference Optimization (GRPO) to update the prefix. Experiments demonstrate that VoiceTTA significantly outperforms existing baselines on rare speech prompts, markedly improving both stylistic fidelity and naturalness of synthesized speech.

Abstract

Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/

Problem

• zero-shot text-to-speech
• voice imitation
• unseen speaking styles
• test-time adaptation
• personalization

Innovation

• test-time adaptation
• reinforcement learning
• zero-shot TTS
• voice imitation
• flow matching