Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids

2026-03-03

AI Summary

This work addresses the challenges of kinematic mismatch and complex contact dynamics in physically coupled interactions between dual humanoid robots by proposing Rhythm, a unified control framework. Rhythm integrates interaction-aware motion retargeting (IAMR), graph-structured reinforcement learning (IGRL), and sim-to-real transfer techniques to establish an end-to-end pipeline that translates human interaction data into coordinated robotic control. The framework enables high-level collaborative behaviors—such as hugging and dancing—that require precise whole-body coordination. These capabilities are successfully demonstrated for the first time on the Unitree G1 physical platform, validating Rhythm’s effectiveness in achieving robust, synchronized interaction in real-world environments.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Realizing interactive whole-body control for multi-humanoid systems is critical for unlocking complex collaborative capabilities in shared environments. Although recent advancements have significantly enhanced the agility of individual robots, bridging the gap to physically coupled multi-humanoid interaction remains challenging, primarily due to severe kinematic mismatches and complex contact dynamics. To address this, we introduce Rhythm, the first unified framework enabling real-world deployment of dual-humanoid systems for complex, physically plausible interactions. Our framework integrates three core components: (1) an Interaction-Aware Motion Retargeting (IAMR) module that generates feasible humanoid interaction references from human data; (2) an Interaction-Guided Reinforcement Learning (IGRL) policy that masters coupled dynamics via graph-based rewards; and (3) a real-world deployment system that enables robust transfer of dual-humanoid interaction. Extensive experiments on physical Unitree G1 robots demonstrate that our framework achieves robust interactive whole-body control, successfully transferring diverse behaviors such as hugging and dancing from simulation to reality.

Problem

• interactive whole-body control
• multi-humanoid systems
• kinematic mismatches
• contact dynamics
• physically coupled interaction

Innovation

• Interactive Whole-Body Control
• Dual Humanoids
• Interaction-Aware Motion Retargeting
• Interaction-Guided Reinforcement Learning
• Sim-to-Real Transfer