ST4VLA: Spatially Guided Training for Vision-Language-Action Models

2026-02-10

AI Summary

This work addresses the challenge that large vision-language models struggle to reliably map high-level instructions to low-level actions in embodied tasks, often due to inconsistent spatial perception and action generation. To overcome this limitation, the authors propose ST4VLA, a two-stage framework: first, it pretrains a spatial localization module via point, bounding box, and trajectory prediction to learn visual-spatial priors; then, during action-oriented fine-tuning, it introduces a spatial prompting mechanism to guide the policy network toward generating actions aligned with spatial targets. This approach enables joint optimization of spatial understanding and action decision-making, achieving significant performance gains on the SimplerEnv benchmark—improving success rates from 66.1 to 84.6 on Google Robot and from 54.7 to 73.2 on WidowX—and demonstrates strong generalization under unseen objects, rephrased instructions, and long-horizon perturbations.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks, where instructions must be transformed into low-level motor actions. We introduce ST4VLA, a dual-system Vision-Language-Action framework that leverages Spatial Guided Training to align action learning with spatial priors in VLMs. ST4VLA includes two stages: (i) spatial grounding pre-training, which equips the VLM with transferable priors via scalable point, box, and trajectory prediction from both web-scale and robot-specific data, and (ii) spatially guided action post-training, which encourages the model to produce richer spatial priors to guide action generation via spatial prompting. This design preserves spatial grounding during policy learning and promotes consistent optimization across spatial and action objectives. Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 ->84.6 on Google Robot and from 54.7 ->73.2 on WidowX Robot, establishing new state-of-the-art results on SimplerEnv. It also demonstrates stronger generalization to unseen objects and paraphrased instructions, as well as robustness to long-horizon perturbations in real-world settings. These results highlight scalable spatially guided training as a promising direction for robust, generalizable robot learning. Source code, data and models are released at https://internrobotics.github.io/internvla-m1.github.io/

Problem

• vision-language-action
• embodied AI
• spatial grounding
• robot learning
• action generation

Innovation

• Spatially Guided Training
• Vision-Language-Action Models
• Spatial Grounding
• Robot Learning
• Multimodal Alignment