Reinforcement Learning-based Receding Horizon Control using Adaptive Control Barrier Functions for Safety-Critical Systems

2024-03-26

arXiv.org

AI Summary

To address the low feasibility and poor performance of Control Barrier Function–Model Predictive Control (CBF-MPC) in safety-critical systems caused by manual tuning, this paper proposes a reinforcement learning (RL)-driven receding-horizon optimization framework. It is the first to embed deep RL algorithms—specifically Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC)—into a hierarchical MPC-CBF architecture, enabling joint adaptive learning of cost function weights and CBF constraint parameters. This approach overcomes the limitations of heuristic parameter tuning while simultaneously ensuring safety guarantees and optimizing control performance. Evaluated in a connected and automated vehicle (CAV) merging scenario, the method significantly improves trajectory smoothness and traffic throughput while reducing infeasibility rates by over 60% compared to hand-tuned baselines. The core contribution is the establishment of the first RL-MPC-CBF bilevel adaptive optimization paradigm, providing a scalable, end-to-end safety-critical control framework for highly reliable autonomous systems.

Technology Category

• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Optimal control methods provide solutions to safety-critical problems but easily become intractable. Control Barrier Functions (CBFs) have emerged as a popular technique that facilitates their solution by provably guaranteeing safety, through their forward invariance property, at the expense of some performance loss. This approach involves defining a performance objective alongside CBF-based safety constraints that must always be enforced. Unfortunately, both performance and solution feasibility can be significantly impacted by two key factors: (i) the selection of the cost function and associated parameters, and (ii) the calibration of parameters within the CBF-based constraints, which capture the trade-off between performance and conservativeness. %as well as infeasibility. To address these challenges, we propose a Reinforcement Learning (RL)-based Receding Horizon Control (RHC) approach leveraging Model Predictive Control (MPC) with CBFs (MPC-CBF). In particular, we parameterize our controller and use bilevel optimization, where RL is used to learn the optimal parameters while MPC computes the optimal control input. We validate our method by applying it to the challenging automated merging control problem for Connected and Automated Vehicles (CAVs) at conflicting roadways. Results demonstrate improved performance and a significant reduction in the number of infeasible cases compared to traditional heuristic approaches used for tuning CBF-based controllers, showcasing the effectiveness of the proposed method.

Problem

• Reinforcement Learning optimizes control parameters
• Adaptive Control Barrier Functions ensure safety
• Model Predictive Control enhances vehicle merging efficiency

Innovation

• Reinforcement Learning-based optimization
• Adaptive Control Barrier Functions
• Model Predictive Control integration