Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning

2026-03-28

AI Summary

This work addresses the inefficiency of existing on-policy reinforcement learning exploration methods, which often overlook the intrinsic value of states and struggle to discover high-reward trajectories. The authors propose a directed exploration mechanism grounded in a differentiable dynamics model, integrating task objectives and physics-informed guidance into the exploration process through analytical policy gradients. This approach represents the first use of analytical policy gradients to drive purposeful exploration, departing from conventional paradigms that rely indiscriminately on entropy maximization or state novelty. Empirical results demonstrate that the proposed framework significantly accelerates policy convergence and enhances final performance on robotic control tasks, exhibiting superior efficiency and stability in exploration and learning.

Abstract

On-policy reinforcement learning (RL) algorithms have demonstrated great potential in robotic control, where effective exploration is crucial for efficient and high-quality policy learning. However, how to encourage the agent to explore the better trajectories efficiently remains a challenge. Most existing methods incentivize exploration by maximizing the policy entropy or encouraging novel state visiting regardless of the potential state value. We propose a new form of directed exploration that uses analytical policy gradients from a differentiable dynamics model to inject task-aware, physics-guided guidance, thereby steering the agent towards high-reward regions for accelerated and more effective policy learning.

Problem

• on-policy reinforcement learning
• exploration
• robotic control
• policy gradient
• directed exploration

Innovation

• analytical policy gradient
• directed exploration
• differentiable dynamics model
• on-policy reinforcement learning
• robotic control