Stabilizing Policy Gradient Methods via Reward Profiling

2025-11-20

AI Summary

High variance in gradient estimation undermines the reliability and convergence speed of policy gradient methods. To address this, we propose a general reward profiling framework that enables selective policy updates driven by statistical confidence, guaranteeing monotonic performance improvement with high probability per update while preserving the original algorithm’s theoretical convergence rate. The framework requires no modification to the underlying policy gradient algorithm and is plug-and-play across diverse policy optimizers. Experiments on eight continuous control benchmark tasks demonstrate an average 1.5× acceleration in convergence speed and up to a 1.75× reduction in return variance, significantly improving learning stability and sample efficiency. Our core innovation lies in coupling high-confidence performance evaluation with dynamic update control—yielding a provably effective, broadly applicable solution for robust policy learning in complex environments.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Policy gradient methods, which have been extensively studied in the last decade, offer an effective and efficient framework for reinforcement learning problems. However, their performances can often be unsatisfactory, suffering from unreliable reward improvements and slow convergence, due to high variance in gradient estimations. In this paper, we propose a universal reward profiling framework that can be seamlessly integrated with any policy gradient algorithm, where we selectively update the policy based on high-confidence performance estimations. We theoretically justify that our technique will not slow down the convergence of the baseline policy gradient methods, but with high probability, will result in stable and monotonic improvements of their performance. Empirically, on eight continuous-control benchmarks (Box2D and MuJoCo/PyBullet), our profiling yields up to 1.5x faster convergence to near-optimal returns, up to 1.75x reduction in return variance on some setups. Our profiling approach offers a general, theoretically grounded path to more reliable and efficient policy learning in complex environments.

Problem

• Addresses unstable reward improvements in policy gradient methods
• Reduces high variance in gradient estimation for faster convergence
• Provides selective policy updates using high-confidence performance estimations

Innovation

• Universal reward profiling framework for policy gradients
• Selective policy updates using high-confidence estimations
• Maintains convergence speed while reducing performance variance