AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance

2026-06-28

AI Summary

Existing physics-based full-body humanoid control methods rely on expensive motion capture systems or hierarchical strategies, limiting their ability to achieve flexible and coordinated motion. This work proposes AnyBody, the first unified full-body control framework capable of driving motion from arbitrary subsets of body keypoints without requiring retargeting, thereby enabling versatile motion tracking and task generalization. By integrating teacher-student distillation, a unit-sphere latent space, a Transformer-based keypoint encoder, and a lightweight residual corrector, AnyBody efficiently supports large-scale motion tracking, freeform teleoperation, and diverse downstream tasks—including walking, aerial writing, and obstacle-aware grasping—significantly enhancing both control flexibility and generalization capability.

Abstract

We present AnyBody, a unified whole-body humanoid controller driven by an arbitrary subset of body keypoints chosen at deploy time. Prior physics-based trackers either rely on expensive full-body motion capture and error-prone trajectory retargeting, which bottleneck scalable data collection and policy learning, or decompose upper- and lower-body control into separate hierarchical representations, sacrificing the coordinated whole-body motions that loco-manipulation requires. We close this gap by learning a single latent motion representation that any keypoint subset can address. To achieve this, we first train a privileged teacher tracker on a large unstructured motion corpus and distill it online into a deterministic encoder-decoder student whose latent space is a unit sphere. We then train a transformer keypoint encoder that admits any subset of body keypoints through masked self-attention, aligning it to the privileged latent. Additionally, we treat the frozen decoder as a motor prior and specialize downstream tasks with a lightweight residual corrector in the latent space. We demonstrate the effectiveness of AnyBody by tracking large-scale human motions from arbitrary keypoint subsets, free-form control, flexibly teleoperating, and learning downstream behaviors including locomotion, in-air writing, and obstacle-reach.

Problem

• whole-body control
• humanoid
• keypoint guidance
• loco-manipulation
• motion tracking

Innovation

• whole-body control
• arbitrary keypoints
• latent motion representation
• transformer encoder
• motion distillation