Predicting When to Trust Vision-Language Models for Spatial Reasoning
This work addresses the systematic errors and unreliable confidence estimation of vision-language models (VLMs) in spatial reasoning tasks, which hinder their deployment in safety-critical applications. The authors propose a novel confidence estimation framework grounded in external geometric verification, eschewing conventional text-based self-evaluation. Their approach integrates four visual signals—object detection, geometric alignment, spatial ambiguity, and internal VLM uncertainty—and combines them via a gradient-boosting model to predict reliability. Evaluated on BLIP-2 and CLIP, the method achieves AUROC scores of 0.674 and 0.583, respectively, representing improvements of 34.0% and 16.1% over baselines. At 60% target accuracy, it attains a coverage rate of 61.9%—more than twice that of the baseline—and boosts scene graph construction accuracy from 52.1% to 78.3%, demonstrating strong cross-architecture generalization.