InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

2026-08-24

AI Summary

本文提出InstructMove基准，通过在包含语义干扰的拾取和放置场景中测试视觉-语言-动作模型对文本指令的依赖性，以解决现有评估方法无法有效验证模型是否真正遵循语言指令的问题。

Abstract

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Problem

• instruction-following
• manipulation
• text-indispensable
• visual saliency
• language grounding

Innovation

• text-indispensable
• instruction-following manipulation
• semantic distractors
• VLA models
• visual shortcuts