VLA Grounder: Language-Conditioning Space Optimization for Black-Box VLA Models

2026-07-05

AI Summary

This work addresses the sensitivity of black-box vision-language-action (VLA) models to phrasing variations in natural language instructions, which leads to unstable task success rates. The study proposes a novel approach that treats language instructions as optimizable variables, reframing human-provided commands into compact, structured prompts enriched with object appearance, spatial relationships, and goal-anchoring cues—without updating the VLA model’s parameters. By leveraging a failure-case-driven command prior and a frozen black-box VLA policy, the method optimizes the language conditioning space via reinforcement learning. This strategy significantly improves performance on instruction-sensitive tasks involving symbolic reasoning and multi-object manipulation, achieving state-of-the-art results on the RL4VLA and VL-Think benchmarks.

Abstract

Vision-Language-Action (VLA) models are commonly treated as end-to-end action policies conditioned on natural-language task descriptions. In practice, however, their behavior often depends sharply on how the instruction is phrased, suggesting that language is not merely a task label but an optimizable conditioning input. We study whether frozen VLA policies can be improved by optimizing language space rather than updating action weights. Our method introduces a language-conditioning space policy that translates a human instruction into a short VLA-grounded command using object appearance, spatial relations, and target-grounding cues. The language-conditioning space policy is initialized with a failure-derived command-space prior and optimized with reinforcement learning from sparse task-completion rewards, while the downstream VLA remains fully frozen. This yields language-conditioning space optimization: RL discovers which VLA-grounded commands best elicit successful behavior from the frozen action policy. Experiments on RL4VLA and VL-Think show that language-conditioning space optimization improves success on instruction-sensitive, symbolic, and multi-object manipulation tasks, demonstrating that language can serve as an optimizable variable for a robot foundation models. Website: https://tttonyalpha.github.io/vla_grounder

Problem

• Vision-Language-Action
• language conditioning
• instruction sensitivity
• frozen policy
• robot foundation models

Innovation

• language-conditioning space optimization
• Vision-Language-Action (VLA)
• frozen policy
• reinforcement learning
• instruction grounding