Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints

2024-11-28

arXiv.org

AI Summary

This work addresses reinforcement learning under almost-sure safety constraints over infinite horizons, motivated by high-reliability applications such as autonomous driving, finance, and resource management. For continuous state-action spaces, we propose a novel policy optimization framework based on dual regularization. To our knowledge, this is the first approach to embed safety constraints directly into a mean-field Wasserstein gradient flow formulation. Under convex regularization, we establish a global exponential convergence guarantee for the resulting dynamics. By constructing a differentiable, bounded approximation of the safety constraint and implementing a particle-based discretization scheme, we prove existence and uniqueness of solutions and derive an explicit convergence rate. Our framework provides the first continuous-policy optimization method for high-dimensional safe RL with rigorous, theoretically grounded convergence guarantees.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Sustainability of Web economics

Abstract

This paper studies reinforcement learning (RL) in infinite-horizon dynamic decision processes with almost-sure safety constraints. Such safety-constrained decision processes are central to applications in autonomous systems, finance, and resource management, where policies must satisfy strict, state-dependent constraints. We consider a doubly-regularized RL framework that combines reward and parameter regularization to address these constraints within continuous state-action spaces. Specifically, we formulate the problem as a convex regularized objective with parametrized policies in the mean-field regime. Our approach leverages recent developments in mean-field theory and Wasserstein gradient flows to model policies as elements of an infinite-dimensional statistical manifold, with policy updates evolving via gradient flows on the space of parameter distributions. Our main contributions include establishing solvability conditions for safety-constrained problems, defining smooth and bounded approximations that facilitate gradient flows, and demonstrating exponential convergence towards global solutions under sufficient regularization. We provide general conditions on regularization functions, encompassing standard entropy regularization as a special case. The results also enable a particle method implementation for practical RL applications. The theoretical insights and convergence guarantees presented here offer a robust framework for safe RL in complex, high-dimensional decision-making problems.

Problem

• Reinforcement learning with almost-sure safety constraints
• Convex regularized objective for continuous state-action spaces
• Exponential convergence guarantees under sufficient regularization

Innovation

• Convex regularization for safe policy optimization
• Mean-field theory with Wasserstein gradient flows
• Exponential convergence under sufficient regularization