A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

2026-07-12

AI Summary

This study addresses the challenge of training a single policy to perform multi-task block-pushing with diverse shapes under sparse rewards and without demonstration data, while achieving zero-shot sim-to-real transfer. To this end, the authors propose a novel approach that integrates reinforcement learning with diffusion policies, introducing a reweighted evidence lower bound (ELBO) loss, inverse curriculum generation, and goal-centric state representations to enhance exploration efficiency. This work presents the first instance of training diffusion-based policies from scratch via reinforcement learning for multi-task robotic manipulation. The resulting policy learns a generalizable strategy in simulation and successfully transfers to the real world without fine-tuning, demonstrating robust zero-shot generalization across varying block shapes, weights, surface frictions, and target positions.

Abstract

Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.

Problem

• diffusion policy
• multi-task manipulation
• block pushing
• zero-shot sim-to-real transfer
• reinforcement learning

Innovation

• diffusion policy
• reinforcement learning
• multi-task manipulation
• zero-shot sim-to-real transfer
• reverse curriculum generation