GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

2024-06-06

IEEE Transactions on Neural Networks and Learning Systems

AI Summary

In safety-critical reinforcement learning (SRL), early data scarcity leads to inaccurate constraint approximation and frequent safety violations. To address this, we propose GenSafe, a general-purpose safety enhancer that— for the first time—formulates safety constraints as a reduced-order Markov decision process (ROMDP). By jointly performing model-order reduction and constraint dimensionality reduction, GenSafe enables lightweight, analytically tractable safety-aware policy correction. As a plug-and-play safety layer, it integrates seamlessly with mainstream SRL algorithms—including CPO and PPO-Lag—without altering the underlying policy architecture. Evaluated across multiple benchmark tasks, GenSafe significantly improves early-stage constraint satisfaction (average +37%) while preserving task performance and incurring minimal deployment overhead. Its design ensures strong generalizability across diverse environments and practical applicability in real-world safety-critical systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Reinforcement Learning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Safe Reinforcement Learning (SRL) aims to realize a safe learning process for Deep Reinforcement Learning (DRL) algorithms by incorporating safety constraints. However, the efficacy of SRL approaches often relies on accurate function approximations, which are notably challenging to achieve in the early learning stages due to data insufficiency. To address this issue, we introduce in this work a novel Generalizable Safety enhancer (GenSafe) that is able to overcome the challenge of data insufficiency and enhance the performance of SRL approaches. Leveraging model order reduction techniques, we first propose an innovative method to construct a Reduced Order Markov Decision Process (ROMDP) as a low-dimensional approximator of the original safety constraints. Then, by solving the reformulated ROMDP-based constraints, GenSafe refines the actions of the agent to increase the possibility of constraint satisfaction. Essentially, GenSafe acts as an additional safety layer for SRL algorithms. We evaluate GenSafe on multiple SRL approaches and benchmark problems. The results demonstrate its capability to improve safety performance, especially in the early learning phases, while maintaining satisfactory task performance. Our proposed GenSafe not only offers a novel measure to augment existing SRL methods but also shows broad compatibility with various SRL algorithms, making it applicable to a wide range of systems and SRL problems.

Problem

• Safe Reinforcement Learning
• Initial Data Scarcity
• Rule Compliance

Innovation

• GenSafe
• Reinforcement Learning
• Safety Compliance