3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

2026-09-08

AI Summary

该研究通过从多视角图像预测3D一致路径点(3DWay)来解决机器人操作中的3D空间定位和视觉-语言推理问题，提高了操作的泛化能力。

Abstract

Intermediate representations are key to bridging the modality gap between generalizable manipulation policies and large-scale pretrained vision-language models (VLMs). Among these, trajectory-based representations compactly represent motion-relevant cues, yet most existing approaches predict trajectories in 2D image space, resulting in intrinsic 3D ambiguity. Moreover, using 2D trajectories with depth still leaves the free-space waypoints ambiguous, limiting reliable 3D reasoning. To address this, we propose predicting 3D consistent waypoints (3DWay) from multi-view images. By reformulating 3D waypoints prediction as generating multi-view consistent 2D waypoints followed by geometric triangulation, we enable explicit 3D motion specification while preserving the strong priors of pretrained VLMs. The predicted waypoints can guide existing VLA models for better generalization or be directly executed on simple tasks. Extensive experiments show that 3DWay substantially improves 3D spatial grounding and vision-language reasoning, demonstrating strong potential for generalizable robot manipulation. Codes will be released at https://github.com/ziqin-h/3DWay.

Problem

• 3D ambiguity
• trajectory prediction
• robot manipulation
• vision-language models
• spatial grounding

Innovation

• 3D Consistent Waypoints
• Multi-view Images
• Geometric Triangulation
• Vision-Language Models