HumDex:Humanoid Dexterous Manipulation Made Easy

2026-03-12

AI Summary

This work addresses the challenge of acquiring high-quality demonstration data for dexterous manipulation in humanoid robots, which is often hindered by limitations in existing teleoperation systems—such as poor portability, severe occlusion, or insufficient precision. To overcome these issues, the authors propose HumDex, a portable teleoperation system that integrates IMU-based full-body motion capture with a calibration-free, learning-based hand retargeting method to enable accurate and easily deployable human motion recording. Furthermore, they introduce a two-stage imitation learning framework comprising pretraining on human demonstrations followed by fine-tuning on robot-collected data, substantially enhancing policy generalization to novel objects, poses, and backgrounds. The system achieves high-fidelity manipulation with minimal data requirements and the complete solution is open-sourced.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This paper investigates humanoid whole-body dexterous manipulation, where the efficient collection of high-quality demonstration data remains a central bottleneck. Existing teleoperation systems often suffer from limited portability, occlusion, or insufficient precision, which hinders their applicability to complex whole-body tasks. To address these challenges, we introduce HumDex, a portable teleoperation system designed for humanoid whole-body dexterous manipulation. Our system leverages IMU-based motion tracking to address the portability-precision trade-off, enabling accurate full-body tracking while remaining easy to deploy. For dexterous hand control, we further introduce a learning-based retargeting method that generates smooth and natural hand motions without manual parameter tuning. Beyond teleoperation, HumDex enables efficient collection of human motion data. Building on this capability, we propose a two-stage imitation learning framework that first pre-trains on diverse human motion data to learn generalizable priors, and then fine-tunes on robot data to bridge the embodiment gap for precise execution. We demonstrate that this approach significantly improves generalization to new configurations, objects, and backgrounds with minimal data acquisition costs. The entire system is fully reproducible and open-sourced at https://github.com/physical-superintelligence-lab/HumDex.

Problem

• humanoid manipulation
• dexterous manipulation
• teleoperation
• demonstration data
• motion tracking

Innovation

• teleoperation
• IMU-based motion tracking
• learning-based retargeting
• imitation learning
• humanoid manipulation