Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

2024-05-04

arXiv.org

AI Summary

Deep reinforcement learning (DRL) lacks per-timestep hard safety guarantees in continuous control tasks. Method: This paper proposes a model-free, provably safe control framework that constructs implicit safe sets and barrier certificates via black-box dynamics queries—without requiring explicit system dynamics—and synthesizes a safety-critical control policy satisfying forward invariance and finite-time convergence. Contribution/Results: It is the first method to provide rigorous theoretical safety guarantees for both continuous- and discrete-time systems. The framework supports parallelizable scaling to high-dimensional systems. Evaluated on the Safety Gym benchmark, it achieves zero safety violations while attaining 95% ± 9% of the cumulative reward of state-of-the-art (SOTA) unsafe methods, thereby simultaneously ensuring safety and competitive performance.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Deep reinforcement learning (DRL) has demonstrated remarkable performance in many continuous control tasks. However, a significant obstacle to the real-world application of DRL is the lack of safety guarantees. Although DRL agents can satisfy system safety in expectation through reward shaping, designing agents to consistently meet hard constraints (e.g., safety specifications) at every time step remains a formidable challenge. In contrast, existing work in the field of safe control provides guarantees on persistent satisfaction of hard safety constraints. However, these methods require explicit analytical system dynamics models to synthesize safe control, which are typically inaccessible in DRL settings. In this paper, we present a model-free safe control algorithm, the implicit safe set algorithm, for synthesizing safeguards for DRL agents that ensure provable safety throughout training. The proposed algorithm synthesizes a safety index (barrier certificate) and a subsequent safe control law solely by querying a black-box dynamic function (e.g., a digital twin simulator). Moreover, we theoretically prove that the implicit safe set algorithm guarantees finite time convergence to the safe set and forward invariance for both continuous-time and discrete-time systems. We validate the proposed algorithm on the state-of-the-art Safety Gym benchmark, where it achieves zero safety violations while gaining $95% pm 9%$ cumulative reward compared to state-of-the-art safe DRL methods. Furthermore, the resulting algorithm scales well to high-dimensional systems with parallel computing.

Problem

• Ensuring safety guarantees in deep reinforcement learning applications
• Synthesizing model-free safe control without explicit dynamics
• Achieving zero safety violations while maintaining high performance

Innovation

• Implicit safe set algorithm for model-free control
• Synthesizes safety index via black-box dynamics queries
• Guarantees finite-time convergence and forward invariance