Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

2026-06-15

AI Summary

This study addresses a critical oversight in the evaluation of vision-language models (VLMs): the erroneous assumption that languages map one-to-one with writing systems, thereby neglecting users of multigraphic languages. To rectify this, we introduce PuMVR, the first multimodal visual reasoning benchmark for Punjabi, encompassing its three major scripts—Gurmukhi, Shahmukhi, and Roman—and conduct a systematic evaluation of ten prominent VLMs on strictly parallel image–text data. We uncover a previously unreported “script gap,” wherein model performance varies by up to 16% across scripts for the same language, and propose Script Consistency Rate (SCR) as a novel metric to quantify script independence. Results reveal SCR as low as 24.8%, demonstrate that visual inputs fail to bridge this gap, and show that cross-script contextual transfer is highly fragile, collectively proving that current “multilingual” VLMs do not genuinely support multigraphic languages.

Abstract

Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.

Problem

• multilingual evaluation
• script consistency
• Vision-Language Models
• multi-script languages
• orthographic gap

Innovation

• Script Consistency
• Multilingual Vision-Language Models
• PuMVR Benchmark
• Script Gap
• Script-agnostic Evaluation