Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization

2026-06-25

AI Summary

Existing vision-language-action (VLA) models struggle to generalize at test time to novel object placements or unseen task instructions, exhibiting limited spatial and task-level generalization. This work proposes a lightweight, model-agnostic injection mechanism that, for the first time, demonstrates the critical role of directly integrating 3D grounding signals into the action head to enhance generalization. The approach computes the relative displacement between 3D points and the gripper using a two-layer MLP and incorporates adaptive layer normalization, enabling efficient signal injection without modifying the backbone architecture or pretraining pipeline. Evaluated on the LIBERO-PRO benchmark, the method boosts the success rate of GR00T-N1.6 from 31.2% to 77.5% under task perturbations and from 28.1% to 60.2% under position perturbations, substantially outperforming current state-of-the-art approaches.

Abstract

Vision-Language-Action (VLA) models leverage large-scale vision-language pretraining for flexible robot manipulation, yet at test time they remain brittle along two axes: spatial generalization, when object positions differ from those seen during training, and task generalization, when a familiar scene is paired with a different language instruction than the one seen in training. A growing family of methods addresses this brittleness by endowing a policy with the spatial and task-aware information such as 2D pixel-coordinate for object localization and placement. However, we find that existing representation through language prompting or visual prompting does not address the limitations; in contrast, exploiting a 3D point-based representation and feeding it directly to the action head leads to substantial improvements-revealing that how the grounding signal is represented and injected into the VLA is the true game changer. Thus, we propose a lightweight, model-agnostic module that represents the grounding signal in 3D, computes its relative displacement to the gripper, and injects the resulting spatial embedding directly into the action head through adaptive layer normalization. The entire module is a two-layer MLP that requires no changes to the VLA backbone or pretraining pipeline. On LIBERO-PRO, our method improves the average success rate of GR00T-N1.6 from 31.2 to 77.5 points under task perturbation and from 28.1 to 60.2 points under position perturbation (gains of 46.3 and 32.1 points). Comparable gains are achieved for $π_{0.5}$ as well, demonstrating that the mechanism is backbone-agnostic. Together, these results support our central finding: given adequate grounding lifted into 3D, injecting it directly into the action head is what unlocks both spatial and task generalization in VLAs-achievable with nothing more than a lightweight module on top of a pretrained backbone.

Problem

• spatial generalization
• task generalization
• Vision-Language-Action models
• grounding signal
• 3D representation

Innovation

• 3D grounding
• action-head injection
• spatial generalization
• task generalization
• vision-language-action (VLA)