STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning

2026-06-29

AI Summary

This work addresses the challenge in real-world robot learning where demonstration and interaction data often contain stalls, errors, and suboptimal behaviors, lacking a reliable frame-level signal to discern task progress. The authors propose STEAM, the first method to enable label-free, frame-level advantage modeling by training an ensemble of temporal offset predictors on expert trajectories. Normalized temporal offsets serve as a self-supervised signal, which is transformed into scalar advantage estimates via the ensemble’s predictive distribution, with conservative evaluation achieved through the ensemble minimum. Integrated with CFGRL policy optimization, STEAM significantly improves policy success rates by 59%, 54.3%, 23%, and 16.2% on real-world tasks including bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place, effectively identifying and suppressing degenerate behaviors.

Abstract

Real-world robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM further improves policy success rate by 59%, 54.3%, 23% and 16.2% over baselines, respectively.

Problem

• robot learning
• heterogeneous data
• frame-level advantage
• demonstrations
• suboptimal behavior

Innovation

• self-supervised learning
• temporal ensemble
• advantage modeling
• real-world robot learning
• frame-level scoring