MobileH2R: Learning Generalizable Human to Mobile Robot Handover Exclusively from Scalable and Diverse Synthetic Data

2025-01-08

AI Summary

To address the challenge of enabling mobile robots to reliably receive objects from humans in large-scale workspaces, this paper proposes a vision-driven human–robot handover learning framework trained exclusively on synthetic data. Methodologically, we introduce (i) a full-body pose-controllable synthetic data generation pipeline; (ii) an automated safe demonstration construction mechanism; and (iii) a novel 4D (3D spatial + temporal) imitation learning paradigm for end-to-end base–manipulator coordinated closed-loop control. Our key contribution is the first demonstration that high-quality, large-scale, and diverse synthetic data alone—without any real human demonstrations—can train a handover policy with strong generalization capability. Both simulation and real-robot experiments show over 15% improvement in handover success rate, validating that synthetic data can effectively replace real annotated data, significantly enhancing model generalization and deployment efficiency.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper introduces MobileH2R, a framework for learning generalizable vision-based human-to-mobile-robot (H2MR) handover skills. Unlike traditional fixed-base handovers, this task requires a mobile robot to reliably receive objects in a large workspace enabled by its mobility. Our key insight is that generalizable handover skills can be developed in simulators using high-quality synthetic data, without the need for real-world demonstrations. To achieve this, we propose a scalable pipeline for generating diverse synthetic full-body human motion data, an automated method for creating safe and imitation-friendly demonstrations, and an efficient 4D imitation learning method for distilling large-scale demonstrations into closed-loop policies with base-arm coordination. Experimental evaluations in both simulators and the real world show significant improvements (at least +15% success rate) over baseline methods in all cases. Experiments also validate that large-scale and diverse synthetic data greatly enhances robot learning, highlighting our scalable framework.

Problem

• Mobile Robotics
• Human-Robot Interaction
• Object Handover

Innovation

• MobileH2R
• Virtual Data Training
• Human-Robot Interaction