TokAN: Accent Normalization Using Self-Supervised Speech Tokens

2026-07-04

AI Summary

This work addresses the challenge of converting non-native accented speech to standard pronunciation without relying on parallel corpora or synthetic supervision, which often degrades output quality and fails to preserve speaker identity. The authors propose an end-to-end accent normalization framework that operates without synthetic supervision, leveraging self-supervised discrete speech tokens. A jointly trained vector-quantized tokenizer extracts shared L1–L2 representations, while an autoregressive encoder–decoder performs token-level conversion from L2 to L1. A duration-aware flow-matching vocoder reconstructs Mel-spectrograms, and the system is further enhanced with a GRPO-based reinforcement learning fine-tuning strategy and speaker embedding conditioning. Evaluated on seven English accents, the method reduces word error rate from 12.40% to 9.23%, significantly outperforming existing baselines and achieving state-of-the-art performance in both accent removal and intelligibility.

Abstract

Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.

Problem

• accent normalization
• non-native speech
• speaker identity preservation
• speech conversion
• L2 to L1

Innovation

• accent normalization
• self-supervised speech tokens
• reinforcement learning
• flow-matching synthesizer
• duration-aware synthesis