Learning-Enhanced Safeguard Control for High-Relative-Degree Systems: Robust Optimization under Disturbances and Faults

2025-01-26

AI Summary

This work addresses nonlinear systems subject to high-relative-degree state constraints, unknown time-varying disturbances, and actuator faults. We propose a reinforcement learning (RL) control framework that jointly ensures safety and performance. Methodologically, the approach integrates high-order differential modeling, control barrier function (CBF)-based safety constraints, and deep RL. Key contributions include: (1) the first rigorous formulation of a high-order reciprocal CBF (HO-RCBF), enabling strict satisfaction of high-relative-degree safety constraints; and (2) a gradient similarity metric coupled with an adaptive gradient regulation mechanism, which enhances policy optimization efficiency while respecting CBF constraints. Simulation results demonstrate that the framework maintains strict safety compliance under severe disturbances and actuator faults. Moreover, it achieves a 32% improvement in closed-loop performance over state-of-the-art safe RL baselines, significantly enhancing robustness and practical applicability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

Merely pursuing performance may adversely affect the safety, while a conservative policy for safe exploration will degrade the performance. How to balance the safety and performance in learning-based control problems is an interesting yet challenging issue. This paper aims to enhance system performance with safety guarantee in solving the reinforcement learning (RL)-based optimal control problems of nonlinear systems subject to high-relative-degree state constraints and unknown time-varying disturbance/actuator faults. First, to combine control barrier functions (CBFs) with RL, a new type of CBFs, termed high-order reciprocal control barrier function (HO-RCBF) is proposed to deal with high-relative-degree constraints during the learning process. Then, the concept of gradient similarity is proposed to quantify the relationship between the gradient of safety and the gradient of performance. Finally, gradient manipulation and adaptive mechanisms are introduced in the safe RL framework to enhance the performance with a safety guarantee. Two simulation examples illustrate that the proposed safe RL framework can address high-relative-degree constraint, enhance safety robustness and improve system performance.

Problem

• Machine Learning
• Control Optimization
• System Safety

Innovation

• High-order Reciprocal Control Barrier Functions
• Gradient Similarity
• Adaptive Adjustment Mechanism