Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis

2026-01-20

AI Summary

Current neural text-to-speech (TTS) systems rely on speaker embeddings to control accent, yet these embeddings conflate linguistic factors such as accent with non-linguistic attributes like timbre and emotion, resulting in poor interpretability and inadequate disentanglement. This work integrates linguistically motivated phonological rules—such as flapping, retroflexion, and vowel correspondences—into neural TTS models and conducts controlled experiments to analyze the interaction between speaker embeddings and rule-based transformations. We introduce the Phoneme Shift Rate (PSR), a novel metric that quantifies the extent to which speaker embeddings preserve or override phonological rules, thereby revealing representational entanglement between accent and speaker identity. Experimental results demonstrate that combining explicit phonological rules with speaker embeddings yields more authentic accents, while embeddings alone often attenuate rule effectiveness, confirming the coupling of accent and speaker characteristics and offering a new evaluation framework for interpretable accent control.

Technology Category

• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Many spoken languages, including English, exhibit wide variation in dialects and accents, making accent control an important capability for flexible text-to-speech (TTS) models. Current TTS systems typically generate accented speech by conditioning on speaker embeddings associated with specific accents. While effective, this approach offers limited interpretability and controllability, as embeddings also encode traits such as timbre and emotion. In this study, we analyze the interaction between speaker embeddings and linguistically motivated phonological rules in accented speech synthesis. Using American and British English as a case study, we implement rules for flapping, rhoticity, and vowel correspondences. We propose the phoneme shift rate (PSR), a novel metric quantifying how strongly embeddings preserve or override rule-based transformations. Experiments show that combining rules with embeddings yields more authentic accents, while embeddings can attenuate or overwrite rules, revealing entanglement between accent and speaker identity. Our findings highlight rules as a lever for accent control and a framework for evaluating disentanglement in speech generation.

Problem

• accented speech synthesis
• speaker embeddings
• phonological rules
• disentanglement
• accent control

Innovation

• phoneme shift rate
• phonological rules
• accented speech synthesis
• speaker embedding disentanglement
• rule-based TTS