VLind-Bench: Measuring Language Priors in Large Vision-Language Models

2024-06-13

arXiv.org

AI Summary

Large Vision-Language Models (LVLMs) are susceptible to language priors—relying excessively on textual cues while neglecting visual content—leading to bias and hallucination on out-of-distribution images. Existing benchmarks fail to disentangle language priors from confounding factors such as commonsense knowledge and core visual understanding. To address this, we introduce VLind-Bench, the first benchmark explicitly designed to quantify LVLMs’ *image blindness*. It establishes a novel language-prior-disentangled evaluation paradigm, leveraging counterfactual image generation, multi-layer validation protocols, and controlled prompt engineering to rigorously isolate visual perception from linguistic and commonsense reasoning. Crucially, it measures over-reliance on textual patterns only after verifying baseline visual and linguistic competence. Extensive experiments across state-of-the-art LVLMs reveal pervasive and significant language priors, exposing systematic hallucination risks rooted in insufficient visual grounding.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Vision-Language Models (LVLMs) have demonstrated outstanding performance across various multimodal tasks. However, they suffer from a problem known as language prior, where responses are generated based solely on textual patterns while disregarding image information. Addressing the issue of language prior is crucial, as it can lead to undesirable biases or hallucinations when dealing with images that are out of training distribution. Despite its importance, current methods for accurately measuring language priors in LVLMs are poorly studied. Although existing benchmarks based on counterfactual or out-of-distribution images can partially be used to measure language priors, they fail to disentangle language priors from other confounding factors. To this end, we propose a new benchmark called VLind-Bench, which is the first benchmark specifically designed to measure the language priors, or blindness, of LVLMs. It not only includes tests on counterfactual images to assess language priors but also involves a series of tests to evaluate more basic capabilities such as commonsense knowledge, visual perception, and commonsense biases. For each instance in our benchmark, we ensure that all these basic tests are passed before evaluating the language priors, thereby minimizing the influence of other factors on the assessment. The evaluation and analysis of recent LVLMs in our benchmark reveal that almost all models exhibit a significant reliance on language priors, presenting a strong challenge in the field.

Problem

• Measure language priors in LVLMs.
• Disentangle language priors from confounding factors.
• Assess biases and hallucinations in LVLMs.

Innovation

• VLind-Bench measures language priors
• Includes counterfactual image tests
• Assesses basic capabilities first