SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model

2026-06-15

AI Summary

This work addresses the challenge of scaling safe reinforcement learning to open-world vision-language-action (VLA) models, where existing approaches rely on costly real-world exploration or handcrafted safety functions. The authors propose SafeDojo, the first world-model-based safe reinforcement learning framework for VLA agents, which introduces an interactive video world model to decouple task rewards from safety costs within imagined trajectories. SafeDojo jointly optimizes these objectives via a constrained variant of GRPO and integrates a ResNet-based task classifier, a lightweight safety head, and chunked action prediction. Evaluated on the SafeLIBERO benchmark, SafeDojo achieves state-of-the-art performance, surpassing the strongest baseline by 8.25 percentage points in Level I safety success rate. Real-world experiments on a Franka robot demonstrate consistently highest average task and safety success rates across five tasks.

Abstract

Safe control is a prerequisite for real-world embodied intelligence, for which safe reinforcement learning has emerged as a promising paradigm. However, existing safe reinforcement learning methods either require costly real-world exploration or depend on hand-crafted safety functions. Neither scales to vision-language-action models deployed in open-world physical environments. We propose SafeDojo, the first model-based safe reinforcement learning framework for vision-language-action policies designed to learn safe actions through world model-based imagination. Specifically, SafeDojo performs online reinforcement learning on top of an interactive video world model. The world model generates action-conditioned future predictions, from which a tailored ResNet success classifier estimates per-step task progress from imagined frames and a lightweight safety head predicts per-step safety costs from latent context together with the proposed action chunk, enabling simultaneous assessment of task execution and trajectory safety. The decoupled task-reward and safety-cost signals are balanced through a Lagrangian-based constrained GRPO objective, enabling coordinated improvement of task success and safety under explicit constraints. On SafeLIBERO, SafeDojo achieves the best aggregate task success, safe success, and execution efficiency among inference-time safety, model-free RL, and model-based RL baselines, with the best average safe-success rate on both levels and an 8.25 percentage-point improvement over the strongest baseline on Level I. Real-world Franka deployment further shows the best average task and safe-success rates across five tasks. Our results position world model-based safe reinforcement learning as a scalable and generalizable path toward safe embodied intelligence.

Problem

• safe reinforcement learning
• vision-language-action models
• open-world environments
• real-world exploration
• hand-crafted safety functions

Innovation

• world model
• safe reinforcement learning
• vision-language-action
• constrained policy optimization
• interactive imagination