ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

2026-06-25

AI Summary

Current vision-language-action (VLA) models lack systematic safety evaluation in embodied intelligence. This work proposes the first unified safety diagnostic benchmark for VLA systems, establishing a 13-category taxonomy spanning interaction, language, and perception. Leveraging the RoboTwin platform, we design 66 safety-critical scenarios across five robot morphologies, systematically manipulating structural, linguistic, and visual variables. We introduce fine-grained process-level metrics—such as cumulative safety cost and risk exposure duration—to enable nuanced assessment. Experiments reveal that even state-of-the-art VLA policies exhibit significant unsafe behaviors; notably, structural and visual perturbations induce risks more readily than linguistic variations, underscoring that safety is tightly coupled with perceptual fidelity, semantic grounding, and control robustness.

Abstract

In embodied intelligence, safety is a prerequisite for reliable robot deployment in the physical world. Current vision-language-action (VLA) models continue to advance toward general-purpose task capability, yet their embodied safety limits remain poorly understood. To address this gap, we introduce ForesightSafety-VLA, a diagnostic benchmark that makes safety the primary evaluation target for VLA systems. We define a 13-category safety taxonomy covering physical interaction safety (Safe-Core), instruction-side safety (Safe-Lang), and perception-side safety (Safe-Vis), and evaluate policies under three controlled dimensions of variation -- scene structure, language command, and visual observation -- so that failure sources can be diagnosed rather than hidden in a single aggregate score. Beyond binary task success, ForesightSafety-VLA measures process-level risk through cumulative safety cost (CC) and risk exposure time (RET), together with a four-quadrant decomposition of safe/unsafe success and failure. We instantiate 66 safety-augmented base scenarios in RoboTwin across 5 embodiments and report results on representative VLA baselines. Across the evaluated baselines, even the strongest policy incurs non-trivial safety cost and unsafe nominal success, while structure and visual variation induce substantially stronger safety degradation than ordinary language variation. These results suggest that embodied safety is tightly coupled to perception, grounding, and control competence rather than being reducible to post-hoc safety filtering alone.

Problem

• Vision-Language-Action Models
• Embodied Safety
• Safety Benchmark
• Robotic Safety
• Diagnostic Evaluation

Innovation

• Vision-Language-Action Models
• Embodied Safety Benchmark
• Safety Taxonomy
• Cumulative Safety Cost
• Risk Exposure Time