Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models

2026-06-14

AI Summary

This study addresses the significant performance degradation of existing vision–language–action models when executing non-English instructions, revealing a pronounced multilingual gap. The work presents the first systematic evaluation of cross-lingual generalization in such models, introducing a multilingual benchmark and employing principal component analysis of representation spaces to demonstrate that performance drops stem from inconsistent cross-lingual representations. To mitigate this issue, the authors propose a multilingual fine-tuning approach based on principal component alignment, which substantially improves task success rates under non-English instructions. Experimental results validate that representational consistency is crucial for effective cross-lingual generalization and significantly narrow the performance gap across languages.

Abstract

Vision-Language-Action models have recently demonstrated promising capabilities in learning generalist robot policies from large-scale multimodal data. However, most existing VLA systems are trained and evaluated primarily with English instructions, leaving their ability to understand and execute instructions in other languages largely unexplored. While the underlying large language models often possess multilingual capabilities, it remains unclear whether these multilingual capabilities transfer to VLAs during training. In this work, we present the first systematic study of multilingual instruction following in VLA models. We first construct multilingual instructions by extending existing benchmarks with translations of their instructions. Using these instructions, we evaluate several representative VLA models across a range of tasks in simulation settings. Our experiments reveal a significant multilingual gap: models trained primarily on English instructions exhibit substantial performance degradation when evaluated on other languages, even when the underlying language backbone is multilingual. We provide several findings and analyses to understand the multilingual gap. Cross-lingual transfer behavior analysis shows that performance drops correlate with both instruction understanding and action execution. Representation analyses suggest that multilingual instruction-caused representation shifts may contribute to the multilingual gap. Motivated by these findings, we further explore strategies to improve multilingual performance in VLAs. We propose a simple yet effective multilingual fine-tuning approach, Multilingual Principal Component Alignment, which leverages Principal Component Analysis to get the principal component subspace and align projected multilingual representations, effectively reducing the multilingual performance gap.

Problem

• Vision-Language-Action models
• multilingual gap
• instruction following
• cross-lingual transfer
• multilingual evaluation

Innovation

• multilingual gap
• vision-language-action models
• cross-lingual transfer
• principal component alignment
• multilingual fine-tuning