HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models

2026-04-14

AI Summary

This work addresses the critical yet overlooked issue of semantic safety in vision-language-action (VLA) models, which may execute hazardous behaviors due to insensitivity to semantic risks and lack systematic evaluation protocols. To this end, we introduce HazardArena—a benchmark comprising over 2,000 assets and 40 tasks—featuring a twin-scenario design that, for the first time, exposes the disconnection between task execution and semantic safety in VLA models. Building on this insight, we propose a training-free safety option layer that integrates either semantic attribute analysis or a vision-language judge to intervene in action selection. This approach significantly reduces unsafe behaviors without compromising task performance, enabling effective, training-agnostic semantic safety enforcement.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision-Language-Action (VLA) models inherit rich world knowledge from vision-language backbones and acquire executable skills via action demonstrations. However, existing evaluations largely focus on action execution success, leaving action policies loosely coupled with visual-linguistic semantics. This decoupling exposes a systematic vulnerability whereby correct action execution may induce unsafe outcomes under semantic risk. To expose this vulnerability, we introduce HazardArena, a benchmark designed to evaluate semantic safety in VLAs under controlled yet risk-bearing contexts. HazardArena is constructed from safe/unsafe twin scenarios that share matched objects, layouts, and action requirements, differing only in the semantic context that determines whether an action is unsafe. We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts. HazardArena includes over 2,000 assets and 40 risk-sensitive tasks spanning 7 real-world risk categories grounded in established robotic safety standards. To mitigate this vulnerability, we propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge, substantially reducing unsafe behaviors with minimal impact on task performance. We hope that HazardArena highlights the need to rethink how semantic safety is evaluated and enforced in VLAs as they scale toward real-world deployment.

Problem

• semantic safety
• Vision-Language-Action models
• action policies
• unsafe outcomes
• semantic risk

Innovation

• semantic safety
• Vision-Language-Action models
• HazardArena
• twin scenarios
• Safety Option Layer