EgoDemoGen: Novel Egocentric Demonstration Generation Enables Viewpoint-Robust Manipulation

2025-09-26

AI Summary

To address the poor robustness of imitation learning under egocentric viewpoint variations, this paper proposes EgoDemoGen—a novel framework that jointly models action retargeting and viewpoint synthesis for the first time. It fine-tunes a pre-trained video generation model via a self-supervised dual-reconstruction strategy, integrating scene video reprojection, robot-exclusive video rendering, and generative inpainting via EgoViewTransfer to enable controllable generation of cross-view paired demonstration data. Evaluated on both simulation and real-robot platforms, EgoDemoGen achieves absolute success rate improvements of +17.7% and +25.8%, respectively, with performance consistently increasing as the proportion of generated data grows. Its core contribution is establishing the first viewpoint-action joint generation paradigm tailored for robotic manipulation, providing a scalable and robustness-enhancing solution for egocentric imitation learning.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Multi-modal Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Imitation learning based policies perform well in robotic manipulation, but they often degrade under *egocentric viewpoint shifts* when trained from a single egocentric viewpoint. To address this issue, we present **EgoDemoGen**, a framework that generates *paired* novel egocentric demonstrations by retargeting actions in the novel egocentric frame and synthesizing the corresponding egocentric observation videos with proposed generative video repair model **EgoViewTransfer**, which is conditioned by a novel-viewpoint reprojected scene video and a robot-only video rendered from the retargeted joint actions. EgoViewTransfer is finetuned from a pretrained video generation model using self-supervised double reprojection strategy. We evaluate EgoDemoGen on both simulation (RoboTwin2.0) and real-world robot. After training with a mixture of EgoDemoGen-generated novel egocentric demonstrations and original standard egocentric demonstrations, policy success rate improves **absolutely** by **+17.0%** for standard egocentric viewpoint and by **+17.7%** for novel egocentric viewpoints in simulation. On real-world robot, the **absolute** improvements are **+18.3%** and **+25.8%**. Moreover, performance continues to improve as the proportion of EgoDemoGen-generated demonstrations increases, with diminishing returns. These results demonstrate that EgoDemoGen provides a practical route to egocentric viewpoint-robust robotic manipulation.

Problem

• Addresses imitation learning degradation from egocentric viewpoint shifts
• Generates novel egocentric demonstrations using action retargeting and video synthesis
• Improves policy robustness across different egocentric viewpoints in manipulation

Innovation

• Generates paired novel egocentric demonstration videos
• Uses generative video repair model EgoViewTransfer
• Fine-tunes pretrained video model with self-supervised strategy