PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control

2026-06-26

AI Summary

This work addresses the challenge of balancing task performance and strict safety constraints in robotic control within safe reinforcement learning. The authors propose PPO-EAL, a novel framework that integrates the exact augmented Lagrangian method into Proximal Policy Optimization (PPO). By combining clipped policy updates with quadratic penalty terms, the approach provides theoretical guarantees for constraint satisfaction, while a momentum-based update rule for Lagrange multipliers enhances dual variable stability. PPO-EAL is the first first-order safe RL method to achieve exact constraint enforcement without requiring excessively large penalty coefficients, offering both convergence guarantees and deployment robustness. Experiments demonstrate superior performance over existing methods across multiple robotic tasks, achieving simultaneous improvements in reward and safety. Notably, the method enables zero-shot sim-to-real transfer for a gear assembly task, significantly boosting success rate and operational robustness.

Abstract

Reinforcement learning (RL) has emerged as a promising solution to accomplish complex robotic control tasks; however, most of the current work ignores the safety requirements. Safe RL seeks to maximize task performance while satisfying explicit physical constraints, but current algorithms struggle to learn the policy efficiently with precise constraint satisfaction. This work proposes PPO-EAL, a novel first-order constrained policy optimization framework that integrates exact augmented Lagrangian optimization into proximal policy optimization for safe robotic control. By combining clipped policy updates with exact quadratic penalty terms, PPO-EAL achieves theoretically grounded constraint enforcement without requiring impractically large penalty factors. A momentum-regulated multiplier update further improves dual-variable stability, reducing constraint oscillation and unsafe behavior while preserving task performance. We provide exactness and convergence analysis under standard stochastic approximation assumptions. Extensive validation across diverse GPU-accelerated robotic benchmarks-including cart-pole balancing, cart-double-pendulum stabilization, 7-DoF Franka end-effector reaching, and quadrupedal locomotion-demonstrates superior safety precision and reward performance compared with state-of-the-art first-order safe RL baselines. Finally, we demonstrate zero-shot sim-to-real deployment in a contact-rich gear assembly task, where PPO-EAL substantially improves task success, reduces peak contact force, and enhances operational robustness. These results establish PPO-EAL as a general and practically deployable safe RL framework for diverse safety-critical robotic systems.

Problem

• Safe Reinforcement Learning
• Constrained Policy Optimization
• Robotic Control
• Safety Constraints
• Physical Constraints

Innovation

• Exact Augmented Lagrangian
• Proximal Policy Optimization
• Safe Reinforcement Learning
• Constraint Satisfaction
• Sim-to-Real Transfer