Exploring Pre-training Benefits on Phoneme Addition through Fine-tuning in Speech Synthesis

2026-06-18

AI Summary

This study addresses the unclear contribution of pretraining to learning unseen phonemes in low-resource text-to-speech synthesis. Through controlled simulations using large language models to generate phoneme-specific utterances and a real cross-lingual transfer setup from English to Japanese, the authors present the first quantitative analysis—under rigorously controlled confounding factors—of how pretraining affects phoneme expansion tasks. Results demonstrate that fine-tuning significantly improves speech naturalness; however, it does not reduce data requirements for achieving high phoneme recognition accuracy, as measured by phoneme error rate (PER). These findings indicate that pretraining primarily enhances the naturalness of synthesized speech rather than improving the sample efficiency of learning novel phonemes.

Abstract

Transfer learning is widely used for low-resource text-to-speech. When the target corpus contains phonemes unseen in pre-training, the model must expand its phoneme inventory during fine-tuning; we call the process "phoneme addition." However, it remains unclear whether the pre-trained ability to generate seen phonemes contributes to this process. This study investigates phoneme addition in two settings: (1) a simulation setup using LLM-generated phoneme-controlled corpora that enables investigation without considering confounding factors, and (2) a real-speech cross-lingual transfer setup (English to Japanese) to validate whether the findings hold in practice. Experiments in both settings showed that while fine-tuning achieved higher naturalness than training from scratch, it required as much or more data to achieve comparable PER for new phonemes. These results indicate that pre-training mainly contributes to naturalness improvement, but offers limited benefit for phoneme addition.

Problem

• phoneme addition
• pre-training
• fine-tuning
• speech synthesis
• transfer learning

Innovation

• phoneme addition
• pre-training
• fine-tuning
• cross-lingual transfer
• speech synthesis