SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

2026-07-09

AI Summary

This work addresses the challenge of minimizing costly falls during real-world robotic reinforcement learning, where safety is paramount and traditional constrained MDP formulations that trade off reward against safety are inadequate. The authors propose an unbiased policy gradient method in which a primary policy is updated exclusively within a safe region, while a deterministic recovery policy takes over outside this region. By employing the score function only at safe timesteps, the approach avoids computing likelihoods for the deterministic recovery policy, thereby eliminating replay bias inherent in mixed-policy methods. Key contributions include the first unbiased policy gradient estimator compatible with deterministic recovery policies, a closed-form value estimate for recovery-triggering states, and a conditional action imitation loss that substantially accelerates credit assignment near safety boundaries. Experiments show 233×, 48×, and 26× reductions in training falls on HalfCheetah, Ant, and Unitree Go1, respectively, achieving final performance comparable to or better than standard PPO—even attaining 80% of the optimal reward in the challenging Ant environment where recovery policies are unreliable.

Abstract

Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.

Problem

• reinforcement learning
• safe exploration
• robotic training
• fall minimization
• recovery interventions

Innovation

• unbiased policy gradient
• recovery intervention
• safe reinforcement learning
• proximal policy optimization
• deterministic recovery policy