From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models

2025-06-11

AI Summary

Existing VLA evaluation benchmarks suffer from narrow task coverage, absence of natural-language instructions, poor reproducibility, and—critically—fail to quantify the practical contribution of VLM pretraining to robotic policy generalization. To address these limitations, we introduce the first unified VLA benchmark comprising 50 simulated tasks, systematically evaluating out-of-distribution (OOD) generalization across both visual inputs and language instructions. Methodologically, we conduct the first quantitative analysis of transfer gains from VLM pretraining, revealing a critical trade-off: fine-tuning on action data degrades VLMs’ general-purpose reasoning capabilities. Through controlled comparisons across diverse architectures (e.g., RT-2, OpenVLA), we find that current VLA models exhibit strong intent understanding and high-level planning but severely lack robust action execution—resulting in pronounced intent-execution misalignment under OOD conditions. Our key contributions include: (1) a standardized, reproducible VLA evaluation paradigm; and (2) open-sourced benchmark code and task suite.

Technology Category

• Intelligent Robots: Other Foundations and Applications
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

One promise that Vision-Language-Action (VLA) models hold over traditional imitation learning for robotics is to leverage the broad generalization capabilities of large Vision-Language Models (VLMs) to produce versatile,"generalist"robot policies. However, current evaluations of VLAs remain insufficient. Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions. Emerging benchmarks for VLAs that incorporate language often come with limited evaluation tasks and do not intend to investigate how much VLM pretraining truly contributes to the generalization capabilities of the downstream robotic policy. Meanwhile, much research relies on real-world robot setups designed in isolation by different institutions, which creates a barrier for reproducibility and accessibility. To address this gap, we introduce a unified probing suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects. We systematically evaluate several state-of-the-art VLA architectures on this suite to understand their generalization capability. Our results show that while VLM backbones endow VLAs with robust perceptual understanding and high level planning, which we refer to as good intentions, this does not reliably translate into precise motor execution: when faced with out-of-distribution observations, policies often exhibit coherent intentions, but falter in action execution. Moreover, finetuning on action data can erode the original VLM's generalist reasoning abilities. We release our task suite and evaluation code to serve as a standardized benchmark for future VLAs and to drive research on closing the perception-to-action gap. More information, including the source code, can be found at https://ai4ce.github.io/INT-ACT/

Problem

• Evaluating generalization of Vision-Language-Action models in robotics
• Assessing VLM pretraining impact on robotic policy performance
• Bridging perception-to-action gap in vision-language-action models

Innovation

• Simulation-based task suite for VLA evaluation
• Systematic VLA generalization capability analysis
• Standardized benchmark for perception-to-action gap