Stability of Control Lyapunov Function Guided Reinforcement Learning

2026-05-03

AI Summary

Existing reinforcement learning (RL) approaches for humanoid robot control often lack theoretical stability guarantees. This work addresses this gap by formulating RL as an optimal control problem and designing a reward mechanism grounded in Control Lyapunov Functions (CLFs). For the first time, it provides rigorous proofs of exponential stability for CLF-guided RL policies in both continuous- and discrete-time systems. By integrating optimal control theory with reinforcement learning, the proposed framework bridges the divide between empirical performance and formal theoretical analysis. Numerical experiments on double integrator and inverted pendulum systems validate the derived theoretical bounds, and the method is successfully applied to generate stable periodic gaits for a humanoid robot.

Abstract

Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice, yet stability analysis of the corresponding control policies is lacking. Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning. A notable example of this is the use of a control Lyapunov function (CLF) to synthesize the reinforcement learning rewards, a technique known as CLF-RL, which has shown practical success. This paper investigates the stability properties of optimal controllers using CLF-RL with the goal of bridging experimentally observed stability with theoretical guarantees. The RL problem is viewed as an optimal control problem and exponential stability is proven in both continuous and discrete time using both core CLF reward terms and the additional terms used in practice. The theoretical bounds are numerically verified on systems such as the double integrator and cart-pole. Finally, the CLF guided rewards are implemented for a walking humanoid robot to generate stable periodic orbits.

Problem

• Reinforcement Learning
• Control Lyapunov Function
• Stability Analysis
• Humanoid Robotics
• Optimal Control

Innovation

• Control Lyapunov Function
• Reinforcement Learning
• Exponential Stability
• Optimal Control
• Humanoid Locomotion