Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

2026-05-28

AI Summary

This study addresses the significant performance gap in knowledge recall exhibited by large language models when operating in low-resource Indian languages and their code-mixed variants, compared to English. To systematically evaluate cross-lingual knowledge consistency, the authors introduce IndicKLAR—a trilingually aligned benchmark covering 18 official Indian languages and 11 code-mixed variants, validated by native speakers. Evaluating nine open-source models across English, native-language, and code-mixed inputs reveals that native-language accuracy lags behind English by approximately 0.50, while code-mixing narrows this gap to about 0.05. The work further proposes novel prompting strategies such as “Translate-in-Thought,” which substantially enhance cross-lingual knowledge consistency.

Abstract

Large language models recall knowledge reliably in English but often fail on the same query posed in a lower-resourced language -- a crosslingual consistency gap that remains underexplored for Indian languages and their code-mixed counterparts. To study this gap, we introduce IndiKLAR, an Indic extension of the KLAR-CLC benchmark covering 18 of the 22 scheduled Indian languages and pairing them with code-mixed variants for 11 widely used language pairs, with native-speaker verification of both monolingual and code-mixed variants for these 11 settings. This three-way alignment offers a unique opportunity to examine how knowledge recall consistency varies across the spectrum of English, code-mixed, and native Indian language inputs. Evaluating across nine open-weight models, we find that the native-language accuracy gap to English can reach $\sim$0.50, while code-mixed inputs close most of it -- bringing performance within $\sim$0.05 of English without any model-level intervention. Motivated by this, we evaluate several prompting strategies that vary in how language conversion is exposed, including a two-stage translate-then-answer setup, a one-stage joint translation-and-answer prompt, and Translate-in-Thought (TinT) -- a single-step strategy in which the model converts the input internally and emits only the final answer. Across the performance trajectory native $\rightarrow$ code-mixed $\rightarrow$ English, we identify a consistent flip point -- the boundary between incorrect and correct prediction -- that lies between the native and code-mixed settings. Interestingly, this holds whether the trajectory is induced by the input surface form or by the model's internal conversion process.

Problem

• cross-lingual consistency
• code-mixed languages
• Indian languages
• knowledge recall
• language models

Innovation

• cross-lingual consistency
• code-mixed languages
• IndicKLAR
• knowledge recall
• prompting strategies