Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planning

2026-06-27

AI Summary

This work addresses the challenges of transferring manipulation skills from human videos to robots, particularly those arising from embodiment discrepancies, scene variations, and feasibility constraints. To this end, the authors propose an object-centric framework for learning interaction priors that decouples human action semantics from robot-specific execution details. By modeling interaction dynamics between objects and integrating constraint-aware task-and-motion planning, the method enables cross-embodiment and cross-scene task generalization without requiring real robot demonstrations. Experimental validation on both a Franka desktop manipulator and the RBY-1 mobile humanoid robot demonstrates that the approach can robustly and feasibly transfer manipulation skills using only human video demonstrations as input.

Abstract

Human videos are a scalable source of supervision for robot manipulation, as they are abundant and naturally capture rich object interactions. However, transferring human demonstrations to robots remains challenging due to embodiment mismatch, scene variation, and robot-specific feasibility constraints. We present Human2Any, a framework for learning reusable object-centric interaction priors from human videos without requiring real-world robot demonstrations in the target task contexts. Human2Any represents manipulation through object-object interaction motion, capturing task-relevant scene changes while abstracting away embodiment-specific details. It composes learned interaction priors with robot-side feasibility reasoning and motion planning, allowing the same human-derived knowledge to adapt to different embodiments, scene geometries, and task contexts. We validate Human2Any across diverse manipulation settings, including real-world experiments on a Franka tabletop setup and an RBY-1 humanoid mobile robot, demonstrating robust interaction-centric manipulation without real-world robot training data. Project website: https://human2any.github.io/.

Problem

• human-to-robot transfer
• embodiment mismatch
• robot manipulation
• feasibility constraints
• scene variation

Innovation

• human-to-robot transfer
• object-centric interaction priors
• compositional planning
• embodiment-agnostic manipulation
• constraint-aware motion planning