Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data

2026-03-08

AI Summary

Current text-to-speech (TTS) systems struggle to model non-native accents, primarily due to the scarcity of accented speech data. This work proposes a novel approach that fine-tunes a multilingual TTS model on native non-English speech to extract task-specific accent vectors that capture distinctive accent characteristics. For the first time, this method enables fine-grained and composable cross-lingual accent control without requiring any accented training data. It supports continuous adjustment of accent strength and synthesis of mixed accents. Experimental results demonstrate that the proposed technique achieves precise and flexible manipulation of multilingual accents, as evidenced by both objective metrics and human evaluations.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet current Text-To-Speech (TTS) systems primarily model American-accented English due limited accented data. We propose \textit{Accent Vector}, a controllable representation that enables accent manipulation in multilingual TTS without requiring accented training data. \textit{Accent Vector} is derived by fine-tuning a TTS system on native speech of a different language (i.e. non-English) and computing task vectors capturing accent characteristics (i.e. in English). By scaling and interpolating the vector, we achieve fine-grained control over accent strength and generate mixed-accent speech. In addition, it generalizes beyond English, enabling accent control across multiple languages. Objective and human evaluations confirm the effectiveness of Accent Vector for fine-grained and compositional accent control.

Problem

• accent manipulation
• multilingual TTS
• accented speech
• non-native speakers
• controllable accent

Innovation

• Accent Vector
• controllable accent manipulation
• multilingual TTS
• task vector
• accent generalization