Learning Process Rewards via Success Visitation Matching for Efficient RL

2026-06-22

AI Summary

Sparse rewards in reinforcement learning lead to severe credit assignment challenges, significantly hindering policy training efficiency. This work proposes a novel mechanism that, without altering the optimal policy, trains a discriminator to distinguish between successful and unsuccessful trajectories and uses it to construct a dense shaping reward. This reward guides the policy to match the state-action visitation distribution of successful trajectories, thereby providing informative learning signals throughout the entire episode. By integrating discriminator-based success visitation matching with policy fine-tuning, the method substantially accelerates training and improves performance in both simulated and real-world robotic manipulation tasks, outperforming baseline approaches that directly optimize sparse rewards.

Abstract

In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.

Problem

• sparse reward
• reinforcement learning
• credit assignment
• process reward
• policy optimization

Innovation

• success visitation matching
• dense process reward
• sparse reward
• reinforcement learning
• policy finetuning