R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies

2026-06-15

AI Summary

This work addresses the limited spatial generalization of imitation learning–based 2D manipulation policies by proposing a real-to-real demonstration augmentation framework. The approach innovatively integrates lightweight 3D editing with 2D video completion to preserve geometric consistency between actions and observations, while circumventing the need for complex 3D reconstruction or simulation-to-reality transfer. Key components include joint 3D editing based on partial point clouds and end-effector trajectories, occlusion-aware projection, and a dense-control image-to-video generation model. Experiments demonstrate that the framework substantially improves the spatial generalization of both 2D diffusion policies and vision-language action policies on tasks involving spatial shifts, validating the effectiveness of each proposed module.

Abstract

Spatial generalization is critical for imitation-learned manipulation policies, but achieving it typically requires scaling demonstrations across diverse object poses, robot configurations, and camera viewpoints. Data augmentation from a few source demonstrations offers a practical alternative to costly real-world collection. Simulation-based augmentation can create controllable variation, but requires complex environment and object setup and may introduce a sim-to-real gap. Recent real-to-real methods avoid these issues by jointly editing 3D observations and action trajectories from real demonstrations, yet they still rely on strong 3D scene parsing and geometry completion, and often produce observations tailored to 3D pointcloud policies rather than RGB-based 2D policies. We propose R2RDreamer, a real-to-real demonstration augmentation framework that preserves the geometric consistency of 3D action-observation editing while moving visual completion to 2D video space. Specifically, R2RDreamer first performs lightweight 3D augmentation by editing incomplete object pointclouds and end-effector trajectories in a shared 3D frame; it then projects the edited scene into masked image-space control videos with occlusion-aware reasoning and uses a dense-control image-to-video model to complete temporally coherent RGB observations. Experiments on spatially shifted manipulation tasks with both 2D diffusion-style policies and vision-language-action policies show that R2RDreamer improves spatial generalization from limited source demonstrations, with analyses validating the contributions of 3D editing, occlusion-aware projection, and video completion.

Problem

• spatial generalization
• data augmentation
• 2D manipulation policies
• real-to-real
• 3D-aware

Innovation

• real-to-real augmentation
• 3D-aware editing
• occlusion-aware projection
• image-to-video completion
• spatial generalization