Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control

2024-12-10

arXiv.org

AI Summary

This work addresses the challenge of simultaneously achieving high-precision upper-body manipulation and robust lower-body locomotion in whole-body control of humanoid robots. We propose a decoupled hierarchical architecture: the upper body employs inverse kinematics and motion retargeting for dexterous manipulation and—novelly—integrates a Prediction Motion Prior (PMP) based on Conditional Variational Autoencoders (CVAEs), which compresses upper-body motion into low-dimensional latent variables serving as conditional inputs to a lower-body PPO-based reinforcement learning policy optimized for gait generation. The lower body thus focuses exclusively on locomotion stability and efficiency. Evaluated in simulation and on a real humanoid platform, our approach enables stable bipedal walking alongside diverse fine-grained manipulations. Upper-body task accuracy significantly surpasses end-to-end RL baselines, overall whole-body task success rate improves by 42%, and robustness against dynamic external disturbances is substantially enhanced.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Humanoid robots require both robust lower-body locomotion and precise upper-body manipulation. While recent Reinforcement Learning (RL) approaches provide whole-body loco-manipulation policies, they lack precise manipulation with high DoF arms. In this paper, we propose decoupling upper-body control from locomotion, using inverse kinematics (IK) and motion retargeting for precise manipulation, while RL focuses on robust lower-body locomotion. We introduce PMP (Predictive Motion Priors), trained with Conditional Variational Autoencoder (CVAE) to effectively represent upper-body motions. The locomotion policy is trained conditioned on this upper-body motion representation, ensuring that the system remains robust with both manipulation and locomotion. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation. With precise upper-body motion and robust lower-body locomotion control, operators can remotely control the humanoid to walk around and explore different environments, while performing diverse manipulation tasks.

Problem

• Decoupling upper-body control from locomotion for precise manipulation.
• Using Predictive Motion Priors (PMP) for robust upper-body motion representation.
• Enhancing humanoid robot stability and manipulation with CVAE-trained features.

Innovation

• Decouples upper-body control using inverse kinematics
• Uses Predictive Motion Priors with CVAE training
• Combines robust locomotion with precise manipulation