Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving

2023-09-24

2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)

AI Summary

To address the limitations of conventional model predictive control (MPC) in autonomous driving motion planning—namely, restricted solution spaces due to convex approximations and the difficulty of balancing real-time performance with global optimality—this paper proposes a safety-enhanced reinforcement learning (RL) and MPC co-optimization framework. Methodologically, it incorporates an energy-function-based safety index constraint and designs state-dependent, online-updated Lagrange multipliers to embed safety requirements into both RL policy optimization and MPC solving, enabling joint safe optimization of reference trajectory generation and local control. Its key contribution is the first integration of a safety index function with an adaptive Lagrange multiplier mechanism, overcoming convex approximation constraints and enabling broader exploration of globally optimal solutions. Evaluated in highway scenarios, the approach achieves a 23.6% improvement in collision avoidance rate and an 18.4% reduction in jerk (trajectory smoothness), while maintaining millisecond-level real-time responsiveness—outperforming baseline MPC and standard safety-aware RL methods.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Model predictive control (MPC) is widely used for motion planning, particularly in autonomous driving. Real-time capability of the planner requires utilizing convex approximation of optimal control problems (OCPs) for the planner. However, such approximations confine the solution to a subspace, which might not contain the global optimum. To address this, we propose using safe reinforcement learning (SRL) to obtain a new and safe reference trajectory within MPC. By employing a learning-based approach, the MPC can explore solutions beyond the close neighborhood of the previous one, potentially finding global optima. We incorporate constrained reinforcement learning (CRL) to ensure safety in automated driving, using a handcrafted energy function-based safety index as the constraint objective to model safe and unsafe regions. Our approach utilizes a state-dependent Lagrangian multiplier, learned concurrently with the safe policy, to solve the CRL problem. Through experimentation in a highway scenario, we demonstrate the superiority of our approach over both MPC and SRL in terms of safety and performance measures.

Problem

• Improving MPC with safe reinforcement learning for motion planning
• Addressing convex approximation limitations in optimal control problems
• Ensuring safety in autonomous driving via constrained reinforcement learning

Innovation

• MPC enhanced with safe reinforcement learning for trajectory planning
• Constrained RL with safety index ensures driving safety
• State-dependent Lagrangian multiplier learned with policy for optimization