Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning

2026-06-23

AI Summary

This work addresses the challenges of goal ambiguity and poor data efficiency in language-guided imitation learning, where end-to-end vision-language-action models often entangle semantic reasoning with spatial control. To overcome this, the authors propose SVP-IL, a novel architecture that explicitly decouples semantic understanding from geometric localization for the first time. It leverages a vision-language foundation model to parse natural language instructions into zero-shot geometric masks, generating lossless spatial visual prompts that are integrated into a continuous action policy via a lightweight feature fusion mechanism. Evaluated on standard benchmarks, the method achieves a success rate of 67.8% and improves performance from 24.0% to 39.5% with only 50–100 demonstrations, demonstrating remarkable data efficiency and robustness on real robotic systems.

Abstract

While end-to-end Vision-Language-Action (VLA) models show promise in robotic manipulation, their monolithic paradigm inherently couples semantic reasoning and spatial control. This creates a severe alignment bottleneck, limiting precise target disambiguation in data-constrained imitation learning. To overcome this, we propose SVP-IL, a decoupled architecture that explicitly extracts spatial visual grounding from the action generation loop. By leveraging vision-language foundation models, we parse instructions into zero-shot geometric masks, translating language into explicit Spatial Visual Prompts (SVP). These priors are injected into a continuous action generator via a lightweight direct feature-level fusion mechanism. This integration provides explicit and uncorrupted spatial gradient guidance while ensuring highly stable optimization under low-data regimes. Extensive experiments demonstrate that SVP-IL significantly outperforms state-of-the-art VLAs and pure visuomotor baselines. Trained on as few as 50 to 100 demonstrations, SVP-IL improves average success rates on highly ambiguous language-conditioned tasks from 24.0% to 39.5%, achieving 67.8% on standard benchmarks. Real-world robotic experiments further validate its robustness and data efficiency in unstructured physical environments.

Problem

• Vision-Language-Action
• spatial grounding
• imitation learning
• semantic disambiguation
• data efficiency

Innovation

• Spatial Visual Prompts
• Decoupled Architecture
• Language-Conditioned Imitation Learning
• Geometric Grounding
• Vision-Language-Action Models