See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

2026-07-13

AI Summary

This work addresses the challenge of cross-view generalization in vision–language–action (VLA) models caused by misalignment between camera and robot coordinate frames. To this end, the authors propose “robot-centric pointmaps,” which encode 3D scene points into image pixels using the robot’s coordinate system. This approach injects dense geometric information into the model while preserving the input structure expected by pretrained 2D architectures. Notably, it is the first method to embed robot-frame 3D geometry into VLA models in an image-based format, enabling viewpoint-invariant action prediction without requiring modifications to existing model architectures. Evaluated on the RoboCasa benchmark, the method significantly enhances the performance of Pi0.5 and SmolVLA, outperforming multiple baselines. Real-robot experiments further demonstrate its superior generalization capability under unseen camera poses.

Abstract

Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.

Problem

• vision-language-action models
• frame mismatch
• robot-centric coordinates
• viewpoint generalization
• 3D coordinate frames

Innovation

• robot-centric pointmaps
• vision-language-action models
• 3D coordinate frame alignment
• cross-viewpoint generalization
• robot action prediction