H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos

2025-12-10

AI Summary

This work addresses the challenge of generating physically plausible robot manipulation videos from unpaired human interaction videos. The proposed method builds upon the Wan 2.2 video diffusion model and introduces in-context fine-tuning alongside a transferable embodied representation: it jointly reconstructs human and robot arm masks while repairing backgrounds, and explicitly injects geometric cues—including position and orientation—to bridge the morphological gap between humans and robots. Crucially, the approach requires only unpaired robot demonstration videos for training, eliminating the need for costly human–robot video alignment. Experiments demonstrate significant improvements in physical plausibility and motion consistency over existing baselines, achieving superior action fidelity and dynamical realism. To our knowledge, this is the first method to empirically validate the scalable distillation of robot manipulation skills from large-scale, unlabeled human video corpora.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/

Problem

• Translate human interaction videos into robot manipulation videos.
• Eliminate need for paired human-robot training data.
• Generate physically grounded, motion-consistent robot actions.

Innovation

• Video-to-video translation without paired human-robot data
• Inpainting and visual cues bridge embodiment gap for transfer
• Fine-tuning video diffusion model for temporal coherence and realism