Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

2026-06-13

AI Summary

This work addresses the limitation of existing large language model (LLM)-driven text-to-speech approaches, which often neglect speaker-specific prosodic patterns, resulting in insufficient style preservation and reduced speaker similarity. To overcome this, the authors propose a novel context-aware autoregressive prosody modeling paradigm that dynamically predicts prosodic features at the syllable level while leveraging historical information from previously synthesized speech to accurately capture the target speaker’s prosodic style. Built upon an LLM architecture, the proposed method significantly enhances prosody learning and speaker similarity across three benchmark datasets, thereby improving both expressiveness and naturalness in personalized speech synthesis.

Abstract

Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.

Problem

• personalized TTS
• speaker similarity
• prosody prediction
• LLM-based TTS
• style learning

Innovation

• dynamic prosody prediction
• LLM-based TTS
• speaker similarity
• prosody learning
• personalized TTS