Curating Demonstrations using Online Experience

2025-03-05

AI Summary

In robot imitation learning, heterogeneous demonstration data—exhibiting varying quality and sparsity—degrades policy generalization: low-quality demonstrations are often imperceptible to human annotators yet significantly reduce test success rates. To address this, we propose Demo-SCORE, an autonomous demonstration filtering framework leveraging real-robot online interaction feedback. Its core innovation lies in using binary success/failure outcomes from real-world roll-outs as an unsupervised evaluation signal—first of its kind—combined with policy classifier training, cross-validation, and uncertainty-aware filtering to jointly optimize simulation-based pre-screening and real-world closed-loop feedback. Experiments demonstrate that policies trained on Demo-SCORE-filtered demonstrations achieve 15–35 percentage-point improvements in real-robot task success rates over full-dataset baselines, enabling automatic discovery and effective utilization of high-quality demonstrations.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Many robot demonstration datasets contain heterogeneous demonstrations of varying quality. This heterogeneity may benefit policy pre-training, but can hinder robot performance when used with a final imitation learning objective. In particular, some strategies in the data may be less reliable than others or may be underrepresented in the data, leading to poor performance when such strategies are sampled at test time. Moreover, such unreliable or underrepresented strategies can be difficult even for people to discern, and sifting through demonstration datasets is time-consuming and costly. On the other hand, policy performance when trained on such demonstrations can reflect the reliability of different strategies. We thus propose for robots to self-curate based on online robot experience (Demo-SCORE). More specifically, we train and cross-validate a classifier to discern successful policy roll-outs from unsuccessful ones and use the classifier to filter heterogeneous demonstration datasets. Our experiments in simulation and the real world show that Demo-SCORE can effectively identify suboptimal demonstrations without manual curation. Notably, Demo-SCORE achieves over 15-35% higher absolute success rate in the resulting policy compared to the base policy trained with all original demonstrations.

Problem

• Heterogeneous robot demonstrations hinder imitation learning performance.
• Unreliable or underrepresented strategies in data reduce test-time effectiveness.
• Manual curation of demonstration datasets is time-consuming and costly.

Innovation

• Self-curation using online robot experience
• Classifier to filter demonstration datasets
• Improved policy success rate by 15-35%