3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

2026-06-30

AI Summary

This work addresses a critical limitation in existing hierarchical Vision-Language-Action (VLA) models, where 2D trajectories generated by high-level planners lack depth information, leading to geometrically distorted guidance for low-level 3D point cloud policies. To overcome this, the authors propose 3D HAMSTER, the first end-to-end metrically reliable 3D trajectory prediction framework within a hierarchical VLA architecture. By integrating a depth encoder and a dense depth reconstruction loss, the model enables vision-language components to directly output accurate 3D trajectories that seamlessly interface with point cloud–driven low-level controllers. Experiments demonstrate that 3D HAMSTER significantly outperforms current 2D-guided approaches and off-the-shelf vision-language models in both simulated and real-world robotic tasks, particularly excelling under appearance variations, novel language instructions, and complex spatial environments.

Abstract

Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation. Recent work in this paradigm uses 2D end-effector trajectories predicted by a Vision-Language Model (VLM) as explicit guidance for a downstream policy. However, state-of-the-art low-level policies operate in 3D metric space on point clouds, and feeding them 2D guidance that lacks depth forces each waypoint to be assigned the depth of whatever scene surface lies beneath it, producing geometrically distorted trajectories. We propose 3D HAMSTER, a hierarchical framework that closes this gap by having the planner directly output metrically reliable 3D trajectories. We augment a VLM with a dedicated depth encoder and a dense depth reconstruction objective to predict 3D waypoint sequences, which are directly integrated into a pointcloudbased low-level policy. Across 3D trajectory prediction, simulation, and real-world manipulation, 3D HAMSTER consistently outperforms proprietary VLMs and 2D-guided baselines, with the largest gains under appearance-altering shifts and unseen language, spatial, and visual conditions. The project page is available at https://davian-robotics.github.io/3D_HAMSTER/.

Problem

• Hierarchical VLA
• 3D trajectory
• depth ambiguity
• robot manipulation
• vision-language models

Innovation

• 3D trajectory prediction
• hierarchical VLA models
• depth-aware vision-language models
• point cloud control
• metrically accurate planning