Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

2025-09-07

AI Summary

This work addresses the scarcity of real-world demonstration data and poor generalization in robot policy learning. We propose a single-example data augmentation method leveraging 3D generative modeling: from one real-world demonstration, we reconstruct the full 3D scene and synthesize multi-view, multi-initial-state virtual trajectories to construct an “imagined space” suitable for omnidirectional policy training. Our approach jointly optimizes 3D generation and reinforcement learning, enabling faithful recovery of object geometry and motion dynamics—and physically consistent data augmentation—from only a few input images. Evaluated on tasks including grasping, drawer opening, and waste sorting, policies trained with our augmented data significantly outperform existing data-augmentation baselines under test conditions far from the original demonstration’s initial state, achieving 27–41% improvements in generalization performance and substantially reducing reliance on large-scale real-world demonstrations.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent 3D generative models, which are capable of generating full object shapes from just a few images, now open up new opportunities in robotics. In this work, we show that 3D generative models can be used to augment a dataset from a single real-world demonstration, after which an omnidirectional policy can be learned within this imagined dataset. We found that this enables a robot to perform a task when initialised from states very far from those observed during the demonstration, including starting from the opposite side of the object relative to the real-world demonstration, significantly reducing the number of demonstrations required for policy learning. Through several real-world experiments across tasks such as grasping objects, opening a drawer, and placing trash into a bin, we study these omnidirectional policies by investigating the effect of various design choices on policy behaviour, and we show superior performance to recent baselines which use alternative methods for data augmentation.

Problem

• Using 3D generative models for robotic dataset augmentation
• Learning omnidirectional policies from single demonstrations
• Enabling task performance from distant initial states

Innovation

• Uses 3D generative models for dataset augmentation
• Learns omnidirectional policies from imagined data
• Reduces demonstrations needed for policy learning