Joycent: Diffusion-based Accent TTS without Accented Phone Prediction

2026-06-15

AI Summary

This work addresses key limitations in existing accented speech synthesis approaches, which typically rely on explicit conversion from standard to accented phonemes—a process prone to error propagation, dependent on paired data, and inadequate for modeling prosodic and other acoustic characteristics. To overcome these challenges, we propose Joycent, an end-to-end accented text-to-speech (TTS) method based on diffusion models that directly generates accented speech from standard phoneme sequences and a reference utterance, without explicitly predicting accented phonemes. Joycent eliminates the conventional two-stage pipeline by integrating diffusion models with conditional layer normalization (CLN) and leverages the WhisAID model to extract disentangled accent representations. This approach preserves speaker identity while significantly enhancing the naturalness of synthesized accents, outperforming current state-of-the-art baselines.

Abstract

Accent text-to-speech (TTS) aims to synthesize speech with target accents. Existing accent TTS systems typically rely on a two-stage pipeline that first converts standard phone sequences into accented phone sequences and then synthesizes accented speech. However, such approaches suffer from error accumulation and require paired standard-accented phone sequence data, which is often limited in practice. Moreover, text-based accented phone representations are insufficient to model acoustic accent characteristics such as prosody and rhythm. In this work, we propose Joycent, a diffusion-based accent TTS model that synthesizes accented speech directly from standard phone sequences and speech references without accented phone prediction. Joycent integrates accent and speaker representations through conditional layer normalization (CLN) in the text encoder. We introduce WhisAID, a Mandarin accent identification model trained on accented Mandarin speech to extract accent representations. Experimental results show that Joycent improves accentedness while preserving speaker identity compared with baseline systems. We release our code and demos at: https://github.com/oshindow/Joycent-code.

Problem

• accent text-to-speech
• accented phone prediction
• error accumulation
• acoustic accent characteristics
• paired phoneme data

Innovation

• diffusion-based TTS
• accent representation
• conditional layer normalization
• accent identification
• reference-based synthesis