MULTITEXTEDIT: Benchmarking Cross-Lingual Degradation in Text-in-Image Editing

2026-05-04

AI Summary

Existing text-image editing benchmarks are predominantly English-centric, limiting their ability to assess semantic correctness and script fidelity in multilingual settings. To address this gap, this work introduces a controlled benchmark spanning 12 languages, 5 visual domains, and 7 edit types, isolating linguistic variables through shared visual bases and human-annotated reference images. The study proposes a novel Language Script Fidelity (LSF) metric and a two-stage evaluation protocol leveraging large vision models, which integrates region-mask alignment with multidimensional assessment—encompassing semantics, pixel-level accuracy, and LSF—to precisely detect fine-grained errors such as missing diacritics and incorrect right-to-left ordering. Evaluations across 12 state-of-the-art systems reveal substantial cross-lingual performance degradation: Hebrew and Arabic exhibit the weakest results, while Dutch and Spanish perform best; models generally preserve layout but frequently distort script-specific details.

Abstract

Text-in-image editing has become a key capability for visual content creation, yet existing benchmarks remain overwhelmingly English-centric and often conflate visual plausibility with semantic correctness. We introduce MULTITEXTEDIT, a controlled benchmark of 3,600 instances spanning 12 typologically diverse languages, 5 visual domains, and 7 editing operations. Language variants of each instance share a common visual base and are paired with a human-edited reference and region masks, isolating the language variable for cross-lingual comparison. To capture script-level errors that coarse text-matching metrics miss, such as missing diacritics, reversed RTL order, and mixed-script renderings, we introduce a language fidelity (LSF) metric scored by a two-stage LVM protocol that first traces the edited target text and then judges it in isolation, reaching a quadratic-weighted \k{appa} of 0.76 against native-speaker annotators. Evaluating 12 open-source and proprietary systems with LSF alongside standard semantic and mask-aware pixel metrics, we find pronounced cross-lingual degradation for every model, largest on Hebrew and Arabic and smallest on Dutch and Spanish, and concentrated in text accuracy and script fidelity rather than in coarse structural dimensions. We also uncover a pervasive semantic and pixel mismatch, where outputs preserve global layout and background fidelity yet distort script-specific forms.

Problem

• cross-lingual degradation
• text-in-image editing
• language fidelity
• script-level errors
• multilingual benchmark

Innovation

• cross-lingual benchmark
• text-in-image editing
• language fidelity metric
• script-level errors
• visual-language models