VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

2026-03-23

AI Summary

Existing vision-language-action (VLA) models struggle to simultaneously achieve high spatial localization accuracy and strong out-of-distribution generalization due to the tight coupling of instruction understanding, spatial reasoning, and motor control within a single forward pass. To address this limitation, this work proposes VP-VLA, a dual-system framework that explicitly decouples high-level task planning from low-level action execution through structured visual prompts—such as crosshairs or bounding boxes. In this architecture, a System 2 planner interprets language instructions and generates visual prompts indicating target locations, while a System 1 controller produces precise actions conditioned on these prompts, further enhanced by a visual localization loss to improve spatial understanding. Evaluated on Robocasa-GR1-Tabletop and SimplerEnv, VP-VLA achieves absolute success rate improvements of 5% and 8.3%, respectively, substantially outperforming baselines including QwenOFT and GR00T-N1.6.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to robotic control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are then overlaid directly onto visual observations as structured visual prompts, such as crosshairs and bounding boxes. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%, surpassing competitive baselines including QwenOFT and GR00T-N1.6.

Problem

• Vision-Language-Action models
• spatial grounding
• robustness
• out-of-distribution
• robotic control

Innovation

• Visual Prompting
• Vision-Language-Action Models
• Dual-System Architecture
• Spatial Grounding
• Robotic Control