Hidden Language Consistency Phenomena in Reasoning LLMs

2026-08-08

AI Summary

This study addresses the limitations of current evaluation practices for multilingual large language models, which overly emphasize accuracy while neglecting how task difficulty influences consistency between input and output languages. Leveraging the PolyMath benchmark, the authors systematically analyze the interplay among accuracy, thought-language consistency (TC), and answer-language consistency (AC) across eight languages and four difficulty levels. They uncover four distinct behavioral patterns in language consistency as a function of task difficulty, identify a previously unreported “language consistency collapse” phenomenon, and demonstrate that quantization methods—such as GPTQ, AWQ, and AutoRound—can affect consistency independently of accuracy. The findings suggest that models may revert to an internal dominant language under high cognitive load to preserve performance, underscoring the necessity of jointly evaluating accuracy, consistency, and task difficulty in multilingual assessments.

Abstract

Multilingual reasoning models are commonly evaluated by whether they arrive at the correct answer, but not by whether they preserve the intended language while reasoning and responding. This omission conceals important multilingual behaviors that emerge as tasks become harder. In this paper, we study task difficulty, task accuracy, thinking-language consistency (TC), and answer-language consistency (AC) across reasoning models using PolyMath benchmark in eight languages and four difficulty levels. We uncover four findings: (1) language consistency exhibits four difficulty-dependent behaviors: output-language consistency remains aligned with input, remains misaligned, degrades gradually, or collapses abruptly. (2) We identify the language consistency breakdown effect, where increasing difficulty can cause a sudden drop in output-language consistency, especially in less strongly represented and non-Latin-script languages. (3) Due to this breakdown effect, accuracy can be preserved or even improved at a harder difficulty level as the model shifts to its internal dominant language. (4) Quantization can improve or degrade output-language consistency independently of its effect on accuracy, with GPTQ and AWQ often outperforming AutoRound under tolerance-based voting with ε = 1.0. These results show that multilingual capability cannot be characterized by accuracy alone; reliable evaluation should jointly consider task accuracy, language consistency, and task difficulty for multilingual benchmarks.

Problem

• language consistency
• multilingual reasoning
• task difficulty
• output-language alignment
• multilingual evaluation

Innovation

• language consistency
• multilingual reasoning
• task difficulty
• quantization
• thinking-language consistency