TWIST: Teleoperated Whole-Body Imitation System

2025-05-05

AI Summary

Current robotic teleoperation systems struggle to achieve coordinated whole-body motion, typically addressing isolated locomotion or manipulation tasks while underutilizing human motion as a full-degree-of-freedom control interface. This paper proposes an end-to-end whole-body teleoperation framework for humanoid robots: reference motion sequences are generated via motion retargeting from real-world marker-based motion capture (MoCap) data; a novel unified homogeneous neural network controller jointly handles manipulation, legged locomotion, walking, and expressive motions; and a hybrid reinforcement learning and behavior cloning (RL+BC) policy is trained with privileged future-frame information to enhance motion tracking fidelity. Evaluated on a physical humanoid robot, the method achieves significantly reduced tracking error, enabling high-precision, low-latency, and highly coordinated whole-body motion imitation across diverse complex tasks.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io

Problem

• Enable whole-body humanoid teleoperation via motion imitation
• Improve tracking accuracy with future frames and MoCap data
• Achieve versatile motor skills with a unified neural controller

Innovation

• Retargeting human motion to humanoid robots
• Combining reinforcement learning and behavior cloning
• Incorporating future motion frames and MoCap data