WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation

2026-06-26

AI Summary

This work addresses the challenge of noisy human teleoperation demonstrations in imitation learning—often containing hesitations and backtracking—by introducing WARP, the first fully self-supervised method for dense relative progress modeling. WARP leverages temporal warping augmentations (variable-speed playback and reversal) on successful trajectories to generate signed, frame-level relative progress targets. It constructs WARP-RM to predict normalized inter-frame time differences, which are aggregated into a dense progress signal. Building upon this, WARP-BC reweights high-advantage action segments during behavior cloning. Requiring no manual annotation of subtask boundaries or absolute progress labels, WARP-BC achieves a 19/20 success rate on a dual-arm T-shirt folding task—even as low-quality data proportions increase—dramatically outperforming standard behavior cloning (2/20) and yielding up to an 18× throughput improvement.

Abstract

Scaling imitation learning requires large datasets, yet human teleoperation inevitably produces mixed-quality demonstrations containing hesitations and recoveries. Prior frame-level progress reward models supervise on absolute temporal progress proxies that suffer from label noise, or require costly human annotations to define subtask boundaries. We present WARP (Warp-Augmented Relative Progress), a novel fully self-supervised algorithm for learning dense, signed relative progress magnitudes directly from successful demonstrations. WARP generates per-frame progress targets via time-warp augmentations of demonstrations (variable playback speeds and reversals) and we train WARP-RM to predict the normalized elapsed time between input frames. Aggregating these predictions across overlapping windows yields a dense frame-level progress signal. We then introduce WARP-BC, which leverages these scalar reward estimates to upweight high-advantage action chunks during behavior cloning, where chunk-level advantage is obtained by aggregating per-frame rewards. We evaluate our approach on a physical bimanual robot system performing a long-horizon deformable object manipulation task: folding T-shirts from a random crumpled start. To evaluate policy robustness against suboptimal data, we construct training datasets of varying quality using episode length as a proxy for teleoperation sub-optimality. As the dataset is widened to admit more inefficiencies, WARP-BC maintains a 19/20 success rate compared to vanilla BC's collapse to 2/20, improving throughput by up to 18x.

Problem

• imitation learning
• data curation
• reward modeling
• demonstration quality
• progress estimation

Innovation

• WARP
• relative progress
• self-supervised reward modeling
• time-warp augmentation
• behavior cloning