Stage-Transition Dense Reward Modeling for Reinforcement Learning

2026-06-30

AI Summary

This work addresses the challenge of sparse and delayed rewards in long-horizon robotic manipulation tasks within reinforcement learning, where manually engineered dense rewards are often impractical. The authors propose STDR, a framework that automatically infers semantic task phases from unstructured expert videos and generates phase-transition goal rewards alongside intra-phase progress feedback—eliminating the need for handcrafted reward functions. STDR innovatively integrates visual reward learning, semantic phase reasoning, out-of-distribution detection, and grasp modulation to enhance reward robustness and prevent reward exploitation. Evaluated across 14 cross-platform manipulation tasks, STDR substantially improves sample efficiency and success rates, matching or surpassing performance achieved with human-designed dense rewards. Real-robot experiments further demonstrate its ability to produce stable, task-aligned reward signals consistent with actual task progression.

Abstract

Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.

Problem

• sparse rewards
• dense reward design
• long-horizon manipulation
• reward shaping
• reinforcement learning

Innovation

• Stage-Transition Reward
• Dense Reward Learning
• Visual Reward Modeling
• Out-of-Distribution Detection
• Robotic Manipulation