Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls

2025-07-23

AI Summary

This study systematically evaluates Visual Entailment (VE) as a diagnostic probe for multimodal language models’ (MLLMs) vision–language understanding, assessing its validity and limitations. Using LLaMA-3.2-11B-Vision, we conduct zero-shot, few-shot (3-shot optimal), and fine-tuning experiments on e-SNLI-VE, augmented with explanation generation, BERTScore-based semantic evaluation, and controlled visual ablation analysis. Results reveal that example ordering substantially affects inference, while excessive contextual examples introduce noise. Fine-tuned accuracy reaches 83.3%, surpassing OFA-X; explanation quality achieves a BERTScore F1 of 89.2%. Critically, under visually restricted conditions, BERTScore shows no significant decline—indicating insufficient visual grounding and overreliance on linguistic priors. This work is the first to empirically demonstrate that VE tasks, without rigorous visual control, risk overestimating MLLMs’ visual grounding capability. It provides empirical evidence and methodological insights for developing more robust multimodal evaluation paradigms.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This study investigates the extent to which the Visual Entailment (VE) task serves as a reliable probe of vision-language understanding in multimodal language models, using the LLaMA 3.2 11B Vision model as a test case. Beyond reporting performance metrics, we aim to interpret what these results reveal about the underlying possibilities and limitations of the VE task. We conduct a series of experiments across zero-shot, few-shot, and fine-tuning settings, exploring how factors such as prompt design, the number and order of in-context examples and access to visual information might affect VE performance. To further probe the reasoning processes of the model, we used explanation-based evaluations. Results indicate that three-shot inference outperforms the zero-shot baselines. However, additional examples introduce more noise than they provide benefits. Additionally, the order of the labels in the prompt is a critical factor that influences the predictions. In the absence of visual information, the model has a strong tendency to hallucinate and imagine content, raising questions about the model's over-reliance on linguistic priors. Fine-tuning yields strong results, achieving an accuracy of 83.3% on the e-SNLI-VE dataset and outperforming the state-of-the-art OFA-X model. Additionally, the explanation evaluation demonstrates that the fine-tuned model provides semantically meaningful explanations similar to those of humans, with a BERTScore F1-score of 89.2%. We do, however, find comparable BERTScore results in experiments with limited vision, questioning the visual grounding of this task. Overall, our results highlight both the utility and limitations of VE as a diagnostic task for vision-language understanding and point to directions for refining multimodal evaluation methods.

Problem

• Evaluating Visual Entailment task reliability for vision-language understanding
• Exploring impact of prompt design and visual information on model performance
• Assessing model's reasoning and visual grounding through explanation evaluations

Innovation

• Uses Visual Entailment task for vision-language understanding
• Explores zero-shot, few-shot, fine-tuning settings
• Employs explanation-based evaluations for reasoning