Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

2024-03-13

arXiv.org

AI Summary

Theoretical characterization of iteration efficiency remains lacking for risk-sensitive reinforcement learning (RL). Method: This paper establishes the first rigorous iteration complexity analysis framework for the risk-sensitive REINFORCE algorithm with an exponential utility function. Contribution/Results: We prove that the algorithm converges to an ε-first-order stationary point within O(ε⁻²) iterations—filling a critical gap in the complexity analysis of risk-sensitive policy gradient methods. Moreover, we theoretically demonstrate that moderate risk aversion can accelerate convergence, challenging the conventional assumption of universal superiority for risk-neutral RL. Empirical evaluation across CartPole, MiniGrid, and Robot Navigation benchmarks confirms that the risk-sensitive variant achieves faster convergence and improved policy stability compared to standard REINFORCE, corroborating our theoretical predictions.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Reinforcement Learning (RL) has shown exceptional performance across various applications, enabling autonomous agents to learn optimal policies through interaction with their environments. However, traditional RL frameworks often face challenges in terms of iteration efficiency and robustness. Risk-sensitive policy gradient methods, which incorporate both expected return and risk measures, have been explored for their ability to yield more robust policies, yet their iteration complexity remains largely underexplored. In this work, we conduct a rigorous iteration complexity analysis for the risk-sensitive policy gradient method, focusing on the REINFORCE algorithm with an exponential utility function. We establish an iteration complexity of $mathcal{O}(epsilon^{-2})$ to reach an $epsilon$-approximate first-order stationary point (FOSP). Furthermore, we investigate whether risk-sensitive algorithms can achieve better iteration complexity compared to their risk-neutral counterparts. Our analysis indicates that risk-sensitive REINFORCE can potentially converge faster. To validate our analysis, we empirically evaluate the learning performance and convergence efficiency of the risk-neutral and risk-sensitive REINFORCE algorithms in multiple environments: CartPole, MiniGrid, and Robot Navigation. Empirical results confirm that risk-averse cases can converge and stabilize faster compared to their risk-neutral counterparts. More details can be found on our website https://ruiiu.github.io/riskrl.

Problem

• Analyzes iteration complexity of risk-sensitive policy gradient methods
• Compares convergence speed of risk-sensitive vs risk-neutral RL algorithms
• Evaluates performance in CartPole, MiniGrid, and Robot Navigation environments

Innovation

• Risk-sensitive REINFORCE with exponential utility
• Iteration complexity analysis for FOSP
• Faster convergence in risk-averse cases