An Empirical Recipe for Universal Phone Recognition

2026-03-30

AI Summary

This study addresses the limited robustness of current speech recognition models in cross-lingual and low-resource settings for phoneme recognition. The authors propose PhoneticXEUS, a framework that leverages self-supervised pre-trained representations, large-scale multilingual data, and a unified phoneme annotation scheme to systematically investigate the impact of data scale, model architecture, and loss functions on multilingual phoneme recognition. For the first time, they quantitatively analyze the influence of these key factors across more than 100 languages and provide an in-depth examination of error patterns. The proposed model achieves state-of-the-art performance with a 17.7% phoneme frame error rate (PFER) on multilingual phoneme recognition and 10.6% PFER on accented English. All code and data are publicly released.

Abstract

Phone recognition (PR) is a key enabler of multilingual and low-resource speech processing tasks, yet robust performance remains elusive. Highly performant English-focused models do not generalize across languages, while multilingual models underutilize pretrained representations. It also remains unclear how data scale, architecture, and training objective contribute to multilingual PR. We present PhoneticXEUS -- trained on large-scale multilingual data and achieving state-of-the-art performance on both multilingual (17.7% PFER) and accented English speech (10.6% PFER). Through controlled ablations with evaluations across 100+ languages under a unified scheme, we empirically establish our training recipe and quantify the impact of SSL representations, data scale, and loss objectives. In addition, we analyze error patterns across language families, accented speech, and articulatory features. All data and code are released openly.

Problem

• phone recognition
• multilingual speech processing
• low-resource languages
• cross-lingual generalization
• pretrained representations

Innovation

• phone recognition
• multilingual speech processing
• self-supervised learning
• empirical ablation
• PFER