Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

2026-08-10

AI Summary

This work addresses a key limitation in existing policy gradient methods for safe reinforcement learning, which often fail to converge precisely to the constraint boundary and instead remain suboptimally within the interior of the feasible region, disregarding the structural insight that optimal policies typically reside on the boundary. To overcome this, we propose Boundary-Searching Policy Gradient (BSPG), the first method to explicitly incorporate the constraint boundary structure into first-order gradient updates. BSPG decomposes the update direction into a tangential component that improves reward while maintaining first-order cost invariance, and a residual-driven normal component that pushes the policy toward active constraint boundaries. The resulting update direction assumes an algebraic Lagrangian form, enabling bidirectional boundary approximation without learning dual variables and theoretically guaranteeing satisfaction of the KKT conditions. On Safety-Gymnasium navigation tasks, BSPG significantly outperforms baselines by achieving higher rewards and tighter adherence to constraint boundaries, with theoretical guarantees that the constraint residual converges to zero at a rate of $O(1/\sqrt{T})$.

Abstract

Safe reinforcement learning maximizes reward subject to safety constraints. For Constrained Markov Decision Processes, the linear-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior. We introduce Boundary-Seeking Policy Gradient (BSPG), a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite-horizon $O(1/\sqrt{T})$ bound, the tangential component is a reward-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence. On a standard Safety-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines.

Problem

• Safe Reinforcement Learning
• Constrained Markov Decision Processes
• Constraint Boundary
• Policy Gradient
• Occupancy Measures

Innovation

• Boundary-Seeking Policy Gradient
• Constrained Markov Decision Processes
• Safe Reinforcement Learning
• Lagrangian Optimization
• Constraint Boundary