Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

2024-07-30

arXiv.org

AI Summary

This work addresses the challenge of achieving robust, multimodal whole-body control for humanoid robots in real-world environments. We propose the Masked Humanoid Controller (MHC), a learning-based framework enabling stable standing, natural bipedal locomotion, and partial- or full-body motion imitation—while maintaining balance and resisting external disturbances. Our key contributions include: (i) the first learning-based multimodal whole-body controller demonstrating successful sim-to-real transfer on the Digit V3 robot; (ii) a state-subset masking mechanism for trajectory tracking, supporting partial trajectory specification and seamless integration with handheld joystick control; and (iii) a behavior-library-driven progressive curriculum learning strategy that unifies motion-capture data, video-based motion retargeting, optimization-generated trajectories, and human-annotated demonstrations. Extensive simulation and real-robot experiments validate MHC’s ability to execute diverse, user-specified commands reliably and in real time.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Multimodal Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The foundational capabilities of humanoid robots should include robustly standing, walking, and mimicry of whole and partial-body motions. This work introduces the Masked Humanoid Controller (MHC), which supports all of these capabilities by tracking target trajectories over selected subsets of humanoid state variables while ensuring balance and robustness against disturbances. The MHC is trained in simulation using a carefully designed curriculum that imitates partially masked motions from a library of behaviors spanning standing, walking, optimized reference trajectories, re-targeted video clips, and human motion capture data. It also allows for combining joystick-based control with partial-body motion mimicry. We showcase simulation experiments validating the MHC's ability to execute a wide variety of behaviors from partially-specified target motions. Moreover, we demonstrate sim-to-real transfer on the real-world Digit V3 humanoid robot. To our knowledge, this is the first instance of a learned controller that can realize whole-body control of a real-world humanoid for such diverse multi-modal targets.

Problem

• Develops a controller for humanoid robots to perform multi-modal tasks.
• Ensures balance and robustness during motion tracking and mimicry.
• Enables sim-to-real transfer for diverse whole-body control tasks.

Innovation

• Masked Humanoid Controller for multi-modal control
• Curriculum training with masked motion imitation
• Sim-to-real transfer on Digit V3 humanoid