Safe-Support Q-Learning: Learning without Unsafe Exploration

2026-04-28

AI Summary

This work addresses the challenge of catastrophic outcomes during exploration in reinforcement learning by proposing a safety-constrained Q-learning framework that strictly avoids visiting unsafe states. The approach enforces a safe set constraint on the behavior policy to guarantee that all trajectories remain within a predefined safe region throughout training. It introduces a two-stage training mechanism that separately optimizes the Q-function and the policy, along with a novel KL-regularized Bellman target that aligns the Q-function with the safe behavior policy. Additionally, a general parameterized policy extraction method is developed, applicable across diverse action spaces and policy classes. Empirical results demonstrate that the method completely eliminates unsafe state visits during training while achieving stable learning, accurate value estimation, and performance comparable to or better than existing baselines.

Abstract

Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the policy induced from the trained Q-values and propose a parametric policy extraction method to approximate the optimal policy. Our approach provides a unified framework that can be adapted to different action spaces and types of behavior policies. Experimental results demonstrate that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than existing baselines.

Problem

• safe reinforcement learning
• unsafe exploration
• safety constraints
• Q-learning
• behavior policy

Innovation

• Safe Reinforcement Learning
• Q-Learning
• Safe Exploration
• KL-Regularized Bellman Update
• Behavior Policy