Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation

2026-06-23

AI Summary

This work addresses a key limitation in existing imitation learning methods, which typically assume that human demonstrations progress monotonically and thus struggle with imperfect trajectories containing errors and subsequent corrections. To overcome this, the authors propose the ReTVL framework, which leverages sparsely annotated retry events as supervision to model the degradation–recovery structure surrounding mistakes. ReTVL jointly performs global progress calibration and local pairwise preference learning to learn an execution-error-sensitive value function, thereby relaxing the monotonicity assumption. This value function is then used to reweight demonstration segments for improved behavioral cloning. Evaluated on real-world robotic manipulation tasks, ReTVL produces finer-grained value estimates than baseline approaches and significantly enhances learning performance from demonstrations of mixed quality.

Abstract

Human demonstrations for robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored. However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. In this work, we propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging retry events as sparse supervision. ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.

Problem

• imitation learning
• value learning
• non-monotonic execution
• demonstration errors
• corrective behaviors

Innovation

• Retry-Supervised Learning
• Value Function Learning
• Imitation Learning
• Non-Monotonic Progress
• Demonstration Reweighting