YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese

2026-07-01

AI Summary

This work addresses the insufficient performance of current large language models in handling Japanese kanji polysemy and phonological understanding, as well as the absence of a systematic evaluation benchmark for these capabilities. To bridge this gap, the authors introduce YOMI-Bench, the first comprehensive benchmark specifically designed to evaluate models on Japanese kanji homograph disambiguation, leveraging linguistic characteristics of the Japanese language. The benchmark comprises four tasks that capture critical dimensions such as polyphone identification and context-dependent pronunciation inference. Experimental results demonstrate that both Japanese-specific open-source models and mainstream commercial large language models consistently underperform on kanji pronunciation generation tasks, underscoring the necessity and value of YOMI-Bench in assessing and advancing Japanese phonological comprehension in language models.

Abstract

We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMI-Bench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings.

Problem

• kanji reading
• phonological understanding
• large language models
• Japanese language
• reading ambiguity

Innovation

• YOMI-Bench
• kanji reading
• phonological understanding
• Japanese LLMs
• benchmark evaluation