Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate

2024-03-01

arXiv.org

AI Summary

Natural policy gradient (NPG) methods lack theoretical justification for reusing historical trajectories, as existing importance sampling techniques ignore trajectory dependencies across policy iterations, precluding rigorous convergence analysis. Method: We propose a novel importance sampling–based trajectory reuse mechanism for NPG and establish, for the first time, its theoretical properties under mild regularity assumptions. Contribution/Results: We rigorously prove that the bias introduced by trajectory reuse asymptotically vanishes, preserving global convergence of the algorithm while significantly accelerating convergence rates. Our analysis integrates natural policy gradients, trust-region constraints, and finite-sample convergence theory. Empirical evaluation on standard reinforcement learning benchmarks—including MuJoCo—demonstrates both improved optimization speed and enhanced stability. This work bridges a critical theoretical gap underlying widely used heuristic trajectory reuse strategies, providing formal guarantees for efficient and reliable policy optimization.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Reinforcement learning provides a mathematical framework for learning-based control, whose success largely depends on the amount of data it can utilize. The efficient utilization of historical trajectories obtained from previous policies is essential for expediting policy optimization. Empirical evidence has shown that policy gradient methods based on importance sampling work well. However, existing literature often neglect the interdependence between trajectories from different iterations, and the good empirical performance lacks a rigorous theoretical justification. In this paper, we study a variant of the natural policy gradient method with reusing historical trajectories via importance sampling. We show that the bias of the proposed estimator of the gradient is asymptotically negligible, the resultant algorithm is convergent, and reusing past trajectories helps improve the convergence rate. We further apply the proposed estimator to popular policy optimization algorithms such as trust region policy optimization. Our theoretical results are verified on classical benchmarks.

Problem

• Efficient reuse of historical trajectories in policy optimization.
• Theoretical justification for importance sampling in policy gradient methods.
• Improving convergence rate using past trajectories in reinforcement learning.

Innovation

• Reuses historical trajectories via importance sampling
• Ensures negligible bias in gradient estimation
• Improves convergence rate in policy optimization