SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

2026-06-18

AI Summary

This work addresses the tendency of vision-language models (VLMs) to overlook subtle, localized visual evidence in evidence-intensive tasks, leading to failure. The authors propose a test-time, plug-and-play intervention that generates question-guided visual “spotlights” and leverages answer span prediction entropy as an internal feedback signal to optimize lightweight, sample-specific interventions for enhanced evidence focus. By introducing low-entropy anchors and an entropy-shaping objective, the method effectively disentangles evidence-based confidence from shortcut-driven collapse, maintaining high-confidence predictions while reducing uncertainty. Test-time adaptation is achieved without retraining via Group Relative Policy Optimization (GRPO). Experiments demonstrate consistent performance gains across diverse VLM architectures and benchmarks, along with significantly improved robustness to visual distractions.

Abstract

Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: \url{https://github.com/YinBo0927/SPOT-E}

Problem

• vision-language models
• evidence grounding
• visual evidence
• answer-span prediction
• test-time intervention

Innovation

• entropy shaping
• visual spotlight
• frozen VLMs
• test-time adaptation
• evidence grounding