Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning

2026-06-30

AI Summary

This work addresses the challenge of disentangling whether vision-language models (VLMs) rely on genuine visual input or linguistic priors for spatial reasoning. To this end, the authors propose a training-free causal control framework termed “grayscale ablation,” which combines image substitution, linear probing, and low-rank editing to systematically evaluate visual grounding across 14 prominent VLMs. The study identifies and categorizes three distinct mechanisms: true grounding, prior bias, and sign inversion—revealing that sign inversion along the depth axis intensifies with model scale. Furthermore, the authors demonstrate that this flaw can be effectively mitigated through simple image rotation or low-rank model editing, yielding substantial improvements in spatial reasoning accuracy.

Abstract

The standard way to read latent knowledge out of a model, a linear probe confirmed by a steering recovery, can systematically overstate what a vision-language model (VLM) actually grounds in the image. We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control: replacing the image with a gray blank. Probes decode the within-axis answer at 73--97% across axes, and a training-free projection lifts a near-chance axis from 59% to 79%, exactly the signature of unlocking latent knowledge. The blank-image arbiter refutes it, revealing three grounding regimes that probing conflates: an axis can be grounded (vision-dependent, correct), a prior (vision-independent, with its decode and its apparent recovery a directional default rather than perception), or, surprisingly, inverted: decodable, causally controllable, but deployed with the wrong sign, so the model scores below chance and the error requires looking. The taxonomy holds across the studied VLMs: in fourteen models spanning six language-model families and 2B--27B, horizontal is grounded, vertical is a prior, and depth is inverted, with the inversion emerging at scale within families. The decode-versus-deploy inversion replicates on seven of eight models across five families, and the minimal edit that re-deploys it varies with geometry: a training-free rotation matches a trained edit on the cleanest model, while distributed inversions need a trained low-rank edit, tracing a per-model correction-complexity spectrum. The cheap, self-calibrating arbiter cleanly separates grounded perception, inverted perception, and prior substitution; we argue it should be a default control for latent-knowledge and steering claims in VLMs.

Problem

• grounding
• spatial reasoning
• vision-language models
• latent knowledge
• causal control

Innovation

• vision-language models
• spatial reasoning
• probing
• causal ablation
• grounding