Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation

2024-03-24

AAAI Conference on Artificial Intelligence

AI Summary

In safe reinforcement learning, gradient conflicts between reward optimization and safety constraints hinder simultaneous achievement of high performance and strict safety compliance. To address this, we propose a gradient-manipulation-based soft-switching policy optimization method. Our approach is the first to systematically characterize the gradient conflict mechanism between reward and safety objectives, and to establish a theoretically grounded co-optimization framework with provable convergence guarantees. We further introduce Safety-MuJoCo, a novel benchmark for safe RL evaluation. The method integrates constrained policy optimization with gradient regularization, enabling performance improvement without compromising safety. Extensive experiments on Safety-MuJoCo and OmniSafe demonstrate that our method consistently outperforms state-of-the-art safe RL algorithms. Notably, it achieves Pareto-improved trade-offs: under high-reward policies, it maintains over 95% constraint satisfaction rates—significantly advancing the frontier of reward-safety balance.

Technology Category

• Constraint Satisfaction and Optimization: Constraint Optimization
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Ensuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we aim to address this conflicting relation by leveraging the theory of gradient manipulation. Initially, we analyze the conflict between reward and safety gradients. Subsequently, we tackle the balance between reward and safety optimization by proposing a soft switching policy optimization method, for which we provide convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. Finally, we evaluate the effectiveness of our method on the Safety-MuJoCo Benchmark and a popular safe benchmark, Omnisafe. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization.

Problem

• Balancing reward and safety in reinforcement learning.
• Addressing conflicting gradients between reward and safety.
• Developing a safe RL framework with gradient manipulation.

Innovation

• Gradient manipulation balances reward and safety.
• Soft switching policy optimization method proposed.
• Safety-MuJoCo Benchmark developed for evaluation.