Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

2026-06-17

AI Summary

This work proposes a training-free, zero-shot long-horizon dexterous manipulation method that accurately translates natural language instructions into executable 3D task plans in novel scenes. Leveraging multi-view RGB images, the approach employs a vision-language model to generate task anchors and 2D keypoints in a reference coordinate frame, then innovatively lifts these to 3D by integrating multi-view triangulation with semantic ray voting. The system further combines atomic action retrieval with task-conditioned grasp region generation to enable complex operations such as grasping, placing, and tool use. Evaluated in real-world settings, the method significantly outperforms single-view RGB-D and fine-tuned vision-language-action (VLA) baselines, successfully achieving zero-shot long-horizon manipulation with unseen objects and in previously unencountered environments.

Abstract

We present a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Rather than training an end-to-end policy, our system uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multi-view fusion. This lifting combines triangulation of view-wise VLM groundings with reference-view ray voting, which searches along a semantic camera ray for geometrically consistent candidates across neighboring views. The resulting 3D keypoints support both pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines. We further demonstrate long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes.

Problem

• zero-shot
• long-horizon manipulation
• dexterous manipulation
• 3D grounding
• vision-language model

Innovation

• zero-shot manipulation
• multi-view 3D grounding
• vision-language model
• dexterous grasping
• tool-use reasoning