UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

2025-05-13

AI Summary

Cross-modal imitation learning is hindered by inherent visual and physical disparities between humans and robots. Method: We propose an unsupervised skill representation learning framework that requires no annotated alignment data. Our approach integrates self-supervised contrastive learning, cross-modal video embedding alignment, implicit skill disentanglement, and policy distillation to learn morphology-agnostic skill representations from large-scale unlabeled human and robot videos. Contribution/Results: To our knowledge, this is the first method enabling fully unsupervised, zero-shot, video-prompted human-to-robot decision-making—without any task-specific fine-tuning or paired demonstrations. We validate its effectiveness in both simulation and real-world robotic platforms: given only a human action video as input, the robot successfully executes unseen tasks with significantly improved action selection accuracy. Our core contribution is bridging the cross-morphology gap by establishing the first generalizable, promptable, unsupervised skill representation paradigm.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Representation Learning for Vision

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts. The project website can be found at: https://kimhanjung.github.io/UniSkill.

Problem

• Bridging human-robot embodiment differences for imitation learning
• Learning cross-embodiment skills without labeled data
• Transferring human video skills to robot policies effectively

Innovation

• Learns embodiment-agnostic skill representations
• Transfers human video skills to robot policies
• Uses large-scale unlabeled cross-embodiment video data