Decoupling Planning and Control for Instructable Agents

2026-08-27

AI Summary

本文提出Instruct-to-Act系统，结合预训练视觉-语言模型的规划能力和世界模型控制器的快速响应能力，通过重新标记策略回放段并联合优化多个目标来解决自主代理在陌生环境中的可靠低延迟动作序列生成问题。

Abstract

Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.

Problem

• vision-language models
• high-level plans
• low-latency action sequences
• world-model controllers
• open-ended task guidance

Innovation

• Decoupling Planning and Control
• World-model Controllers
• Instruction-tuned Vision-Language Models (VLMs)
• Behavior Cloning
• Synthetic Instructions