When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias

2026-04-19

AI Summary

This work identifies and formally characterizes an information bias in vision-language models (VLMs) when employed as evaluators—namely, their tendency to favor responses that are more informative while neglecting alignment with the visual content. To address this issue, the authors propose BIRCH, a novel evaluation paradigm that incorporates an image-text consistency correction mechanism. BIRCH first rectifies factual inaccuracies in candidate answers with respect to the image and then performs comparative assessment based on the corrected versions, thereby steering the VLM’s judgment toward faithfulness to visual evidence rather than mere informativeness. Experimental results across multiple models and benchmarks demonstrate that BIRCH effectively mitigates information bias—reducing it by up to 17%—and substantially improves evaluation performance, with gains of up to 9.8%.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often pays limited attention to the image when making decisions. Instead, they often blindly favor the more informative answer, even when they can recognize it conflicts with the image content. We call this problem informativeness bias, which significantly undermines judge reliability. To address it, we propose BIRCH (Balanced Informativeness and CoRrectness with a Truthful AnCHor), a judging paradigm that first corrects inconsistencies with the image content in candidate answers, and then compares the answers against this corrected version. This shifts the judge's focus from informativeness to image-grounded correctness. Experiments on multiple models and benchmarks show that BIRCH reduces informativeness bias by up to 17%, resulting in performance gains of up to 9.8%. Our work reveals an overlooked but fundamental flaw in current VLM-as-a-Judge systems and highlights the need for more principled designs.

Problem

• informativeness bias
• VLM-as-a-Judge
• vision-language models
• judge reliability
• image-grounded correctness

Innovation

• informativeness bias
• VLM-as-a-Judge
• BIRCH
• image-grounded correctness
• truthful anchor