RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation

2026-06-20

AI Summary

This work addresses the challenge of reward design in reinforcement learning for robotic manipulation, where sparse rewards provide weak supervision, handcrafted dense rewards suffer from poor generalization, and existing progress-based rewards are often misled by visually plausible but physically infeasible states. To overcome these limitations, the authors propose RARM, a lightweight visual comparator that generates dense, perceptually grounded progress-based rewards from a single successful demonstration. RARM leverages a temporally contrastive model pretrained on unlabeled videos to achieve task- and platform-agnostic reward modeling and incorporates a confidence-gated mechanism to suppress uncertain matches, thereby avoiding spurious positive signals. Evaluated across nine simulated and four real-world manipulation tasks, RARM significantly improves success rates, demonstrating particularly strong performance on long-horizon tasks such as cloth folding.

Abstract

Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.

Problem

• reward design
• robot manipulation
• reinforcement learning
• progress-based reward
• long-horizon tasks

Innovation

• Reference-Anchored Reward Model
• progress-based reward
• contrastive temporal learning
• confidence-gated reward
• demonstration-free reward design