Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

2026-08-15

AI Summary

This study addresses the performance degradation of multilingual multimodal large language models in non-English visual reasoning by identifying the "Ghost Anchor" phenomenon, which induces modality asynchrony. To mitigate this, we propose ANCHOR, a framework employing an active visual anchoring mechanism that accelerates early visual semantic generation to effectively rectify cross-modal alignment discrepancies. Integrating mechanistic intervention analysis with specialized training strategies, our approach significantly outperforms existing baselines across multiple benchmarks. It achieves robust reasoning in both zero-shot and fine-tuning scenarios for non-English languages, establishing a novel paradigm for enhancing the cross-cultural generalization capabilities of multilingual MLLMs.

Abstract

Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the \textbf{Ghost Anchor} phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose \textbf{ANCHOR}, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.

Problem

• Multilingual MLLMs
• Modality Asynchrony
• Ghost Anchor
• Visual Reasoning Degradation
• Cross-lingual Alignment

Innovation

• Ghost Anchor
• Modality Asynchrony
• ANCHOR Framework
• Proactive Visual Anchoring
• Multilingual MLLMs