Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models

2026-06-24

AI Summary

This work addresses the significant yet underexplored impact of dialectal variation on the performance of large language models in spoken dialogue systems. To systematically evaluate dialect comprehension, the authors propose a novel metric—dialect robustness, defined as the ratio of model performance on dialectal versus standard language inputs—and introduce the first cross-modal (speech and text) evaluation framework for assessing both speech language models (SLMs) and text-based large language models (LLMs). Experimental results demonstrate that the dialect robustness of SLMs is heavily dependent on their underlying LLMs and can be substantially improved through dialect-specific data training and fine-tuning of speech encoders. This study uncovers intrinsic connections in cross-modal dialect understanding and offers effective strategies for enhancing multimodal models’ adaptability to linguistic diversity.

Abstract

Dialogue systems based on large language models (LLMs) have advanced significantly in recent years. However, dialectal variation remains a major challenge, particularly for systems that process spoken input. LLM-based speech language models (SLMs), which integrate LLMs with speech processing components, show promise for spoken language tasks, yet their ability to comprehend dialects has not been sufficiently studied. Moreover, it remains unclear how the dialectal understanding of the base LLM affects SLM performance. This study investigates the dialectal robustness of both LLMs and SLMs using Japanese dialects as a test case. We define robustness as the ratio of performance on dialectal versus standard inputs, enabling fair comparisons. Our experiments show that SLM robustness correlates with that of their text-based counterparts. Furthermore, training with dialectal data and fine-tuning the speech encoder each improves robustness in SLMs.

Problem

• dialect robustness
• large language models
• speech language models
• Japanese dialects
• spoken language understanding

Innovation

• dialect robustness
• speech language models
• large language models
• Japanese dialects
• speech encoder fine-tuning