Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

2026-07-07

AI Summary

This work addresses the susceptibility of goal-conditioned world models to “instruction leakage” in spatial relation understanding—where models directly transcribe answers from instruction text rather than reasoning based on environmental perception. The paper formally defines this issue, highlighting that explicit inclusion of answers in instructions leads to spuriously high accuracy. To mitigate this, the authors propose a goal-agnostic dynamics modeling mechanism that removes goal instructions during forward dynamics learning and applies supervision only at the readout stage. Experiments demonstrate that the proposed approach achieves consistent relation understanding accuracy of 0.88 regardless of instruction presence, substantially outperforming leaky baselines whose performance drops sharply from 0.90 to 0.27 when instructions are withheld. These results validate the method’s effectiveness in enabling genuinely environment-driven grounding of spatial relations.

Abstract

Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking $0.90$ relation-readout accuracy, yet this is \emph{instruction transcription}, not perception. Withholding the goal collapses it to chance ($0.90\!\to\!0.27$, three seeds) and a counterfactual instruction makes the predicted anchors follow the \emph{false} instruction $94.5\%$ of the time (true scene $2.3\%$; $N{=}256$). Tested across three settings and a within-task ablation, our central claim characterizes the confound: \textbf{instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are.} Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names \emph{referents} does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the \emph{read} path, recovering genuine, instruction-independent grounding ($0.88$, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.

Problem

• instruction leakage
• spatial relations
• world models
• grounding
• goal-conditioned prediction

Innovation

• instruction leakage
• world model
• spatial grounding
• goal-free dynamics
• relation readout