Provably Safe, Yet Scalable Reinforcement Learning

2026-06-12

AI Summary

Existing provably safe reinforcement learning methods suffer from the curse of dimensionality and excessive conservatism due to their reliance on explicitly constructing control-invariant sets. This work proposes the PS2-RL framework, a two-stage architecture enabling safe policy learning in high-dimensional systems. First, a backup policy is trained using a safety-reachability value function to implicitly generate control-invariant sets online. Then, a differentiable projection layer is introduced to end-to-end optimize the main policy under safety constraints induced by the backup policy. By avoiding explicit computation of control-invariant sets, PS2-RL is compatible with any underlying reinforcement learning algorithm and demonstrates provable safety, high performance, and scalability in robotic tasks with state dimensions up to 10, significantly overcoming the dimensional and performance limitations of existing approaches.

Abstract

Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.

Problem

• Safe Reinforcement Learning
• Provably Safe
• Scalability
• Control-Invariant Sets
• Constraint Satisfaction

Innovation

• provably safe reinforcement learning
• control-invariant sets
• backup policy
• differentiable projection
• safe-arrival value function