VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation

2024-09-19

AI Summary

Existing VLA models are predominantly evaluated in constrained, manually designed scenarios, leaving their generalization and robustness under real-world deployment insufficiently and systematically examined. To address this, we propose VLATest—the first automated fuzz testing framework specifically designed for vision-language-action models. VLATest employs programmatically synthesized scenes, multi-factor controllable perturbations (e.g., illumination changes, viewpoint shifts, distractor objects, and instruction variations), and cross-model consistency analysis to rigorously stress-test model behavior. Experiments across seven state-of-the-art VLA models reveal an average task success rate drop exceeding 40% in diverse, complex scenarios, clearly exposing critical robustness bottlenecks. This work establishes the first reproducible, scalable benchmark for evaluating VLA model reliability, providing a foundational methodology for model diagnosis, improvement, and safe deployment.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The rapid advancement of generative AI and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model's performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.

Problem

• Evaluating robustness of Vision-Language-Action models in diverse scenarios
• Assessing performance limitations of current VLA models for robotics
• Investigating factors affecting VLA model reliability in manipulation tasks

Innovation

• Fuzzing framework for VLA model testing
• Empirical study on seven VLA models
• Analysis of robustness-impacting factors