Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study

2026-06-20

AI Summary

This study systematically evaluates the performance of large language models (LLMs) on Japanese grapheme-to-phoneme (G2P) conversion to enhance the controllability and robustness of text-to-speech (TTS) systems. Two prompting strategies are proposed: an analytical mode integrating morphological analysis with rule-based kana transcription, and a direct mode performing end-to-end kana prediction. Over thirty LLMs are benchmarked on a manually annotated dataset of 3,000 sentences and compared against conventional G2P tools. Results demonstrate that LLMs significantly outperform traditional methods, with the best model achieving a kana character error rate of 0.52%—substantially lower than the 1.03% attained by standard tools—and yielding superior pronunciation quality when integrated into TTS pipelines compared to end-to-end TTS. This work further reveals, for the first time, the critical influence of model scale, version, and Japanese-specific training on G2P accuracy, and validates the efficacy of the analytical approach combined with rule-based post-processing.

Abstract

Grapheme-to-phoneme (G2P) conversion is essential for controllable and robust text-to-speech, and large language models (LLMs), with broad linguistic knowledge, offer a promising approach. We benchmarked over 30 LLMs on Japanese G2P, comparing them with conventional morphological analyzers on 3000 manually annotated sentences. We evaluated two prompting strategies: a parse mode, where the LLM performs morphological analysis followed by rule-based kana conversion, and a direct mode, where the LLM directly predicts kana readings. The results show that model size, version, and Japanese-specialized training are key factors, with the best LLMs achieving kana character error rate below 0.52\% vs. the best conventional tool (1.03\%). Parse mode outperforms direct mode for most models, as rule-based post-processing relieves the LLM of handling complex pronunciation rules. We also show that feeding LLM-predicted kana into a kana-input TTS yields better pronunciation than end-to-end TTS.

Problem

• Grapheme-to-Phoneme Conversion
• Large Language Models
• Japanese TTS
• Kana Prediction
• Text-to-Speech

Innovation

• grapheme-to-phoneme
• large language models
• Japanese G2P
• prompting strategies
• kana-based TTS