Safe Reinforcement Learning using Ideas from Model Predictive Control

2026-07-08

AI Summary

This work addresses the challenge that reinforcement learning often fails to strictly satisfy hard safety constraints during exploration, risking irreversible damage to physical systems. To overcome this limitation, the authors propose a general safety framework that integrates deep reinforcement learning (DRL) with model predictive control (MPC). The approach leverages a known system dynamics model to construct an offline feasible set of state-action pairs and employs an online safety filter that projects the agent’s actions in real time onto this set, thereby providing formal safety guarantees throughout both training and deployment. By combining the safety assurances of MPC with the adaptive capabilities of DRL, the method enables safe exploration, stable policy convergence, and hardware protection, as demonstrated on a nonlinear single-degree-of-freedom experimental platform.

Abstract

Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring strict, hard safety constraints during the active learning phase. In real-world physical systems, violating mechanical limits can cause irreversible damage, necessitating that exploration remains strictly within safe operational regions. We propose a generalized framework that combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). Using a mathematical model of the system dynamics, offline MPC computations define a feasible state-action space, representing all safe combinations of system states and control inputs that guarantee constraint satisfaction. During training and deployment, the RL agent's instantaneous actions are projected onto this globally verified feasible set via a safety filter. We systematically evaluate our generalized approach on a non-linear 1-DoF laboratory testbed, demonstrating successful exploration and stable policy convergence on physical hardware.

Problem

• Safe Reinforcement Learning
• Hard Safety Constraints
• Cyber-Physical Systems
• Model Predictive Control
• Exploration Safety

Innovation

• Safe Reinforcement Learning
• Model Predictive Control
• Safety Filter
• Feasible State-Action Space
• Cyber-Physical Systems