CLAMP: Constrained Decoding for Vision-Language Embodied Planning

2026-09-08

AI Summary

针对视觉-语言模型生成不可执行计划的问题，提出CLAMP框架，通过场景证据约束解码过程，确保生成的行动计划符合实际环境和安全要求。

Abstract

Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.

Problem

• embodied planning
• vision-language models
• executable actions
• scene evidence
• constraint-grounding

Innovation

• Constrained Decoding
• Multimodal Constraint-Grounding Framework
• Hidden Markov Model (HMM)
• Scene Evidence
• Frozen VLM Planner