OmniTrack: General Motion Tracking via Physics-Consistent Reference

2026-02-27

AI Summary

This work addresses the challenge that humanoid robots often fail to faithfully imitate human motions due to morphological mismatches and data noise, which can produce physically infeasible reference trajectories—such as those involving floating or interpenetration—thereby compromising tracking stability and generalization. To resolve this, the authors propose the first two-stage framework that explicitly decouples physical feasibility from general motion tracking: a privileged universal policy first generates physically consistent trajectories in simulation that respect the robot’s dynamics, and a subsequent universal control policy is trained to track these trajectories. Integrating physics-constrained optimization, reinforcement learning, and online teleoperation, the system achieves hour-long stable execution on real hardware, successfully performing high-difficulty maneuvers like backflips and cartwheels while enabling robust, human-style online control.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Motion & Tracking

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Learning motion tracking from rich human motion data is a foundational task for achieving general control in humanoid robots, enabling them to perform diverse behaviors. However, discrepancies in morphology and dynamics between humans and robots, combined with data noise, introduce physically infeasible artifacts in reference motions, such as floating and penetration. During both training and execution, these artifacts create a conflict between following inaccurate reference motions and maintaining the robot's stability, hindering the development of a generalizable motion tracking policy. To address these challenges, we introduce OmniTrack, a general tracking framework that explicitly decouples physical feasibility from general motion tracking. In the first stage, a privileged generalist policy generates physically plausible motions that strictly adhere to the robot's dynamics via trajectory rollout in simulation. In the second stage, the general control policy is trained to track these physically feasible motions, ensuring stable and coherent control transfer to the real robot. Experiments show that OmniTrack improves tracking accuracy and demonstrates strong generalization to unseen motions. In real-world tests, OmniTrack achieves hour-long, consistent, and stable tracking, including complex acrobatic motions such as flips and cartwheels. Additionally, we show that OmniTrack supports human-style stable and dynamic online teleoperation, highlighting its robustness and adaptability to varying user inputs.

Problem

• motion tracking
• humanoid robots
• physical feasibility
• reference motion artifacts
• generalizable control

Innovation

• physics-consistent motion
• decoupled tracking framework
• privileged generalist policy
• trajectory rollout
• humanoid robot control