Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty

2026-04-28

AI Summary

This work addresses the challenge of ensuring safety during reinforcement learning in high-dimensional systems with unknown dynamics, where existing safety filtering methods are often overly conservative or ineffective. The authors propose Dyna-SAuR, a novel algorithm that integrates uncertainty-aware dynamics modeling within a Dyna-style reinforcement learning framework. With minimal prior knowledge, Dyna-SAuR jointly trains a control policy and a scalable safety filter, while leveraging the learned model to guide safe exploration and dynamically expand the set of feasible states. Experimental results on CartPole and MuJoCo Walker tasks demonstrate that the method reduces training failure rates by two orders of magnitude compared to state-of-the-art approaches, substantially improving both safety and sample efficiency.

Abstract

Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertainty-aware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.

Problem

• safe reinforcement learning
• safe exploration
• uncertainty-aware dynamics
• safety filter
• high-dimensional systems

Innovation

• Safety Augmented Reinforcement Learning
• Uncertainty-aware Dynamics Model
• Safe Exploration
• Dyna-style RL
• Safety Filter