A Multimodal Dataset for Visually Grounded Ambiguity in Machine Translation

2026-05-03

AI Summary

This study addresses the challenge of semantic mistranslation in multimodal machine translation caused by linguistic ambiguity, which existing datasets fail to adequately cover under open-domain scenarios. To bridge this gap, the authors introduce VIDA—the first vision-dependent ambiguity dataset tailored to real-world translation settings—comprising 2,500 expert-annotated samples requiring visual disambiguation. They further propose a Chain-of-Thought Supervised Fine-Tuning (CoT-SFT) strategy and a fine-grained disambiguation evaluation metric leveraging a large language model as a discriminator. Experimental results demonstrate that CoT-SFT significantly improves disambiguation accuracy on both in-distribution and out-of-distribution ambiguities while preserving overall translation quality, thereby confirming the strong generalization capability of the approach across diverse ambiguity types.

Abstract

Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks that provide supportive evidence for the role of vision, we observe substantial issues in data quality and a mismatch with translation scenarios. Moreover, existing ambiguity-oriented evaluations are not well suited to broader ambiguity types in open-ended translation. To address these limitations, we present VIDA (Visually-Dependent Ambiguity), a dataset of 2,500 carefully curated instances in which resolving an annotated ambiguous source span requires visual evidence. We further propose Disambiguation-Centric Metrics that use an LLM-as-a-judge classifier to verify whether annotated ambiguous expressions are resolved correctly at the span level. Experiments with two state-of-the-art Large Vision Language Models under vanilla inference, supervised fine-tuning (SFT), and our chain-of-thought SFT (CoT-SFT) show that while SFT improves overall translation quality, CoT-SFT yields more consistent gains in disambiguation accuracy, especially on out-of-distribution subsets, indicating a stronger generalization for resolving diverse ambiguity types.

Problem

• multimodal machine translation
• ambiguity resolution
• visually grounded ambiguity
• disambiguation
• open-ended translation

Innovation

• multimodal machine translation
• visually grounded ambiguity
• disambiguation-centric metrics
• chain-of-thought SFT
• VIDA dataset