LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

2026-06-22

AI Summary

Existing Vision-Language-Action (VLA) models lack systematic evaluation of operational safety under stringent constraints and are vulnerable to physical collisions and semantic misinterpretations. This work proposes the first parameterized safety benchmark, featuring procedurally generated, highly stochastic safety-critical scenarios and a key-pose-driven data generation pipeline that synthesizes a large-scale dataset of 19,664 collision-free demonstrations. Leveraging this benchmark, we conduct a systematic evaluation of ten prominent VLA models, revealing for the first time an inherent tension between model generalization and safety. Our analysis shows that training with higher trajectory diversity improves safety, yet task success remains limited by the quality of trajectory synthesis and semantic alignment. We further provide a comprehensive failure mode analysis to elucidate the underlying challenges.

Abstract

Despite the impressive manipulation capabilities of Vision-Language-Action (VLA) models, their operational safety under strict constraints remains largely unverified. To address this, we introduce a parametric safety benchmark to procedurally generate safety-critical scenarios with comprehensive stochasticity. To overcome the scalability bottlenecks of human teleoperation, we develop a novel keypose-driven data generation pipeline. Leveraging this infrastructure, we curate a large-scale dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization. We then conduct a systematic cross-paradigm evaluation of eight VLA and two embodied foundation models. Our analysis reveals a critical generalization-safety tension: although high-diversity training fosters safer trajectories, task success remains fundamentally bottlenecked by sub-optimal trajectory synthesis and semantic misalignment. By providing a scalable pipeline, a robust dataset, and profound failure-mode insights, LIBERO-Safety establishes a crucial foundation for developing safe and reliable VLA models.

Problem

• Vision-Language-Action models
• safety benchmark
• physical safety
• semantic safety
• embodied AI

Innovation

• parametric safety benchmark
• keypose-driven data generation
• domain randomization
• trajectory synthesis
• semantic alignment