VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

2026-06-29

AI Summary

This work addresses a key limitation in perception-driven humanoid loco-manipulation research—the scarcity of large-scale, synchronized datasets pairing egocentric vision, natural language instructions, and full-body motion trajectories. To overcome this, the authors propose an end-to-end framework that first reconstructs real indoor environments using 3D Gaussian Splatting to automatically generate large-scale, unannotated vision–language–kinematics (VLK) triplets. A policy network is then trained to predict short-horizon full-body motion trajectories from visual and linguistic inputs, which are executed on a physical robot via a whole-body tracking controller. Experiments on the Unitree G1 humanoid demonstrate successful execution of navigation and single-object manipulation tasks, validating that interaction data synthesized from reconstructed scenes can effectively facilitate sim-to-real transfer.

Abstract

Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation. Project Website: https://vision-language-kinematics.github.io/

Problem

• humanoid loco-manipulation
• vision-language-kinematics
• egocentric perception
• whole-body motion
• sim-to-real

Innovation

• vision-language-kinematics
• 3D Gaussian Splatting
• synthetic data generation
• humanoid loco-manipulation
• sim-to-real transfer