Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation

2025-06-13

AI Summary

Existing Vision-Language Translation (VLT) research lacks systematic multilingual evaluation, and current datasets suffer from significant limitations in semantic and cultural fidelity. To address these issues, we propose three key contributions: (1) AibTrans—the first human-verified, semantically and culturally faithful multilingual OCR correction parallel dataset; (2) DA Score—a density-aware evaluation framework that substantially improves robustness in complex visual contexts; and (3) a balanced multilingual fine-tuning strategy that boosts BLEU scores for low-resource languages by up to +4.2 without degrading general multimodal capabilities. Our experiments span 17 mainstream LVLMs and LLMs, establishing the first standardized VLT benchmark. This benchmark reveals critical insights: strong OCR dependency, divergent generation versus reasoning behaviors across models, and negative cross-lingual transfer effects induced by high-resource language fine-tuning.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Vision-Language Translation (VLT) is a challenging task that requires accurately recognizing multilingual text embedded in images and translating it into the target language with the support of visual context. While recent Large Vision-Language Models (LVLMs) have demonstrated strong multilingual and visual understanding capabilities, there is a lack of systematic evaluation and understanding of their performance on VLT. In this work, we present a comprehensive study of VLT from three key perspectives: data quality, model architecture, and evaluation metrics. (1) We identify critical limitations in existing datasets, particularly in semantic and cultural fidelity, and introduce AibTrans -- a multilingual, parallel, human-verified dataset with OCR-corrected annotations. (2) We benchmark 11 commercial LVLMs/LLMs and 6 state-of-the-art open-source models across end-to-end and cascaded architectures, revealing their OCR dependency and contrasting generation versus reasoning behaviors. (3) We propose Density-Aware Evaluation to address metric reliability issues under varying contextual complexity, introducing the DA Score as a more robust measure of translation quality. Building upon these findings, we establish a new evaluation benchmark for VLT. Notably, we observe that fine-tuning LVLMs on high-resource language pairs degrades cross-lingual performance, and we propose a balanced multilingual fine-tuning strategy that effectively adapts LVLMs to VLT without sacrificing their generalization ability.

Problem

• Evaluating multilingual vision-language translation performance systematically
• Addressing dataset limitations in semantic and cultural fidelity
• Improving translation quality metrics for varying contextual complexity

Innovation

• Introduces AibTrans dataset with OCR-corrected annotations
• Benchmarks 17 LVLMs/LLMs across diverse architectures
• Proposes Density-Aware Evaluation with DA Score