Improving Value Estimation Critically Enhances Vanilla Policy Gradient

2025-05-25

AI Summary

This work investigates why modern policy gradient algorithms (e.g., TRPO, PPO) outperform vanilla policy gradient (PG). We identify value function estimation accuracy—not trust-region constraints—as the primary determinant of performance differences. By simply increasing the number of value network updates per iteration (i.e., strengthening TD-error minimization), vanilla PG achieves significantly improved stability and sample efficiency. Building on the Monte Carlo policy gradient framework, we introduce multi-step TD advantage estimation and batched policy updates. Our method matches or exceeds PPO’s performance across continuous-control benchmarks (e.g., MuJoCo), while exhibiting greater hyperparameter robustness, enhanced training stability, and reduced tuning complexity. The core contribution is the rigorous empirical and conceptual demonstration that high-fidelity value estimation is central to policy gradient success—and that this insight enables a minimal, highly effective PPO alternative.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.

Problem

• Enhancing value estimation improves vanilla policy gradient performance
• Challenging trust regions as key to policy improvement in RL
• Increasing value update steps boosts robustness and effectiveness

Innovation

• Enhancing value estimation accuracy improves performance
• Increasing value update steps boosts robustness
• Simplified vanilla policy gradient matches PPO