Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers

2026-09-09

AI Summary

该研究通过结合生成的视频与基于仿真的手-物体交互跟踪，解决了可靠参考动作缺乏的问题，提高了模拟训练中的成功率和实际操作中的抓取多样性。

Abstract

Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular, we propose a method that enables scalable reference generation by HOI reconstruction with minimal manual intervention and successfully grounds more than 1,500 generated videos in simulation, achieving success rates over 25 percentage points higher than those of baselines during simulation-based training. In real-world closed-loop experiments, it achieves diverse grasps, including functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking. Videos and code are available at https://boyuan-an.github.io/GALATEA/.

Problem

• hand-object interaction
• simulation-based training
• scalable reference generation

Innovation

• HOI tracking
• generated videos
• simulation-based grounding
• scalable reference generation
• multi-object manipulation