Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

2026-06-18

AI Summary

Existing multilingual vision-language models (VLMs) implicitly assume a “one language, one script” paradigm, overlooking the needs of users of multi-script languages such as Punjabi and thereby introducing significant performance disparities due to orthographic differences. This work proposes the first evaluation framework tailored for multi-script languages, introducing the PuMVR benchmark comprising 375 culturally grounded multimodal reasoning tasks spanning Punjabi’s three primary scripts: Gurmukhi, Shahmukhi, and Roman. We introduce Script Consistency Rate (SCR) as a core metric and, through cross-script comparisons and Chain-of-Thought analyses, reveal substantial performance gaps across scripts for identical semantic content: accuracy differences reach up to 16% across ten prominent models, with SCR as low as 24.8%. Notably, visual inputs fail to mitigate this orthographic bias. Our study challenges the prevailing single-script evaluation paradigm and offers a new perspective on fairness in multilingual VLMs.

Abstract

Current Vision-Language Models (VLMs) are celebrated for their multilingual capabilities, yet they operate under a flawed assumption: that one language corresponds to a single writing system. This overlooks billions of users of multi-script languages like Punjabi, Serbian, Hindi-Urdu, Kurdish, among many others, for whom a model's capability may be fractured by orthographic bias. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), the first benchmark designed to quantify script-dependent bias through 375 culturally grounded image-reasoning tasks across Punjabi's three active scripts (Gurmukhi, Shahmukhi, Roman). Evaluating 10 state-of-the-art VLMs, we expose a substantial Script Gap: models frequently solve visual puzzles in one script while failing identical tasks in another, with accuracy deltas reaching 16% and Script Consistency Rates (SCR) as low as 24.8%. Crucially, visual input boosts absolute performance but does not close this gap, the relative bias persists. Our analysis suggests reasoning patterns show limited cross-script transferability, and Chain-of-Thought pathways diverge based on script alone. We propose SCR as a core metric for script-agnostic evaluation, challenging current multilingual assessment paradigms and providing a framework for equitable AI.

Problem

• orthographic bias
• multilingual VLMs
• multi-script languages
• script gap
• script consistency

Innovation

• orthographic bias
• multilingual VLMs
• script consistency
• PuMVR benchmark
• cross-script transferability