Nightmare Dreamer: Dreaming About Unsafe States And Planning Ahead

2026-01-08

arXiv.org

AI Summary

This work addresses the challenge of ensuring safety in reinforcement learning when agents rely solely on image observations—a setting where safety guarantees are difficult to maintain, thereby limiting real-world applicability. The paper proposes a novel world model–based safe reinforcement learning algorithm that, for the first time, integrates dreamer-style safety state prediction with model-based planning. By learning latent dynamics and forecasting potential safety risks, the method embeds safety constraints directly into action planning. Evaluated on image-only Safety Gymnasium tasks, the approach achieves nearly 20× higher sample efficiency compared to model-free baselines while reducing safety violations to near zero, effectively balancing learning efficiency and safety.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Safety and Robustness
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement Learning (RL) has shown remarkable success in real-world applications, particularly in robotics control. However, RL adoption remains limited due to insufficient safety guarantees. We introduce Nightmare Dreamer, a model-based Safe RL algorithm that addresses safety concerns by leveraging a learned world model to predict potential safety violations and plan actions accordingly. Nightmare Dreamer achieves nearly zero safety violations while maximizing rewards. Nightmare Dreamer outperforms model-free baselines on Safety Gymnasium tasks using only image observations, achieving nearly a 20x improvement in efficiency.

Problem

• Reinforcement Learning
• Safety
• Robotics
• Safe RL
• Safety Violations

Innovation

• model-based reinforcement learning
• safe reinforcement learning
• world model
• safety violation prediction
• planning