WorldSample: Closed-loop Real-robot RL with World Modelling

2026-07-02

AI Summary

This work addresses the high interaction cost and inefficient single-trajectory data collection that hinder real-world robot reinforcement learning. The authors propose WorldSample, a framework that establishes a real-to-synthetic closed-loop system: a posterior world model trained on real interaction data generates high-fidelity visual synthetic transitions, while a Policy Rhythm Learning (PRL) mechanism dynamically schedules the use of real and synthetic samples. By preserving visual realism and effectively integrating multi-source data, the approach substantially improves both training efficiency and model fidelity. Experiments on contact-rich and high-precision manipulation tasks demonstrate a 28% increase in policy success rate and a 59% reduction in required real-world interaction steps; additionally, the world model achieves a 19.4 dB gain in PSNR and a 0.47 improvement in SSIM.

Abstract

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

Problem

• real-robot reinforcement learning
• high interaction cost
• physical rollouts
• data efficiency
• world modelling

Innovation

• WorldSample
• real-robot reinforcement learning
• world modeling
• Policy-Paced Learning
• synthetic data augmentation