LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

2026-08-12

AI Summary

Existing confidence-based scoring methods struggle to effectively evaluate the consistency between outputs of large vision-language models (LVLMs) and their input images, particularly failing to detect visual hallucinations. To address this limitation, this work proposes LookBack—a training-free response scoring method that, for the first time, quantifies visual grounding as token-level retrospective scores. By integrating token likelihood with visual attention mechanisms, LookBack measures the degree to which each output token relies on the image content. This approach overcomes the constraints of conventional text-confidence-only metrics and significantly outperforms existing baselines across four benchmarks and three model architectures. Notably, it achieves substantial performance gains in Best-of-N selection tasks while incurring negligible computational overhead.

Abstract

Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.

Problem

• LVLM hallucination
• response scoring
• visual grounding
• confidence metrics
• image-text alignment

Innovation

• LookBack
• visual reference usage
• LVLM hallucination
• response scoring
• vision-language models