CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

2026-06-12

AI Summary

This work addresses the challenge in safe reinforcement learning where delayed constraint corrections often cause policy oscillations near safety boundaries and prolonged constraint violations. To mitigate these issues, the authors propose Constraint-Sensitive Policy Optimization (CSPO), which incorporates local constraint sensitivity into first-order gradient updates by leveraging the signed shortest distance to guide policy correction. This approach preserves the Karush–Kuhn–Tucker (KKT) solution of the original constrained optimization problem while effectively suppressing oscillations and accelerating safe recovery. Built upon a first-order primal-dual framework, CSPO seamlessly integrates with deep reinforcement learning algorithms. Empirical evaluations on navigation and locomotion tasks demonstrate that CSPO significantly outperforms state-of-the-art methods in terms of safety recovery speed, reward retention, and constraint-satisfaction performance.

Abstract

Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods

Problem

• Safe Reinforcement Learning
• Constrained Markov Decision Processes
• Primal-Dual Methods
• Safety Constraints
• Constraint Violation

Innovation

• Constraint-Sensitive Policy Optimization
• Safe Reinforcement Learning
• Primal-Dual Methods
• Constraint Violation Recovery
• KKT Conditions