GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech

2026-07-02

AI Summary

This work addresses the challenge of inaccurate pronunciation of rare proper nouns, loanwords, and technical terms in zero-shot text-to-speech synthesis. It proposes a fine-grained pronunciation control method that introduces, for the first time, a word-level pronunciation guidance mechanism within a neural encoder-decoder language model. The approach extracts pronunciation cues from short audio samples of target words and precisely aligns them to corresponding positions in the text prompt using the model’s built-in phonetic tokenizer. Additionally, a voice conversion strategy is incorporated during training to effectively decouple the prosody of the prompt speech from the target speaker’s voice characteristics. Experimental results across five languages demonstrate a 22–39% reduction in phoneme error rate on target words, significantly higher pronunciation accuracy in human listening tests compared to existing baselines and open-source systems, and consistently high voice similarity and naturalness.

Abstract

We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.

Problem

• pronunciation control
• zero-shot text-to-speech
• proper nouns
• loanwords
• technical terms

Innovation

• pronunciation conditioning
• zero-shot TTS
• neural codec language modeling
• voice disentanglement
• grafted reference audio