Iterative Self-Learning for Expressive Text-to-Speech Synthesis

2026-08-16

AI Summary

This study addresses the scarcity and high annotation cost of explicit expressive labels in emotional speech synthesis by proposing the first semi-supervised iterative self-learning framework tailored for this domain. The core innovation lies in an Invert-Classify invertible classifier that recovers discrete labels from unlabeled speech by inverting a frozen generative model, coupled with a pseudo-labeling and retraining mechanism to enable iterative refinement. Experimental results demonstrate that this approach significantly improves both pseudo-label accuracy and the expressiveness of synthesized speech under low-resource conditions. Notably, the method achieves performance comparable to fully supervised models, effectively mitigating the data bottleneck inherent in emotional text-to-speech systems.

Abstract

Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.

Problem

• Expressive Text-to-Speech
• Label Scarcity
• Semi-supervised Learning
• Low-resource TTS

Innovation

• Iterative Self-Learning
• Expressive Text-to-Speech
• Invert-Classify
• Pseudo-labeling
• Low-resource TTS