Risk-Aware Safe Reinforcement Learning for Control of Stochastic Linear Systems

2025-05-14

AI Summary

This paper addresses safe control of stochastic discrete-time linear systems. We propose a risk-aware safe reinforcement learning (RL) framework that jointly learns a data-driven RL controller and a risk-constrained safety controller, enabling real-time performance–safety trade-offs via noise-robust piecewise affine interpolation. Our contributions are threefold: (1) a novel co-learning mechanism for dual controllers, eliminating conservative safety interventions and unstable equilibrium points; (2) an efficient linear programming solver based on scalar-variable optimization, drastically reducing computational complexity; and (3) expansion of the controlled invariant set while simultaneously decreasing both the probability and variance of safety violations. Simulation results demonstrate that the method achieves high-confidence safety and control performance while significantly reducing the amount of data required for safety verification.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper presents a risk-aware safe reinforcement learning (RL) control design for stochastic discrete-time linear systems. Rather than using a safety certifier to myopically intervene with the RL controller, a risk-informed safe controller is also learned besides the RL controller, and the RL and safe controllers are combined together. Several advantages come along with this approach: 1) High-confidence safety can be certified without relying on a high-fidelity system model and using limited data available, 2) Myopic interventions and convergence to an undesired equilibrium can be avoided by deciding on the contribution of two stabilizing controllers, and 3) highly efficient and computationally tractable solutions can be provided by optimizing over a scalar decision variable and linear programming polyhedral sets. To learn safe controllers with a large invariant set, piecewise affine controllers are learned instead of linear controllers. To this end, the closed-loop system is first represented using collected data, a decision variable, and noise. The effect of the decision variable on the variance of the safe violation of the closed-loop system is formalized. The decision variable is then designed such that the probability of safety violation for the learned closed-loop system is minimized. It is shown that this control-oriented approach reduces the data requirements and can also reduce the variance of safety violations. Finally, to integrate the safe and RL controllers, a new data-driven interpolation technique is introduced. This method aims to maintain the RL agent's optimal implementation while ensuring its safety within environments characterized by noise. The study concludes with a simulation example that serves to validate the theoretical results.

Problem

• Risk-aware RL control for stochastic linear systems
• Data-driven safety without high-fidelity models
• Balancing RL performance and safety guarantees

Innovation

• Risk-aware safe RL controller combined with stabilizing controller
• Piecewise affine controllers for larger invariant sets
• Data-driven interpolation for RL and safe controller integration