The influence of visual and linguistic cues on ignorance inference in Vision-Language Models (VLMs)

2025-02-13

AI Summary

This study investigates visual-language models’ (VLMs’) capacity to derive ignorance inferences—pragmatic implicatures signaling speaker uncertainty—from multimodal contexts. We examine how VLMs jointly process visual cues and linguistic modifiers (bare numerals, superlatives, comparatives) to trigger such inferences. Using a truth-judgment task, we systematically evaluate GPT-4o and Gemini 1.5 Pro across precisely and approximately specified visual contexts. Our work introduces the first orthogonal experimental design that independently manipulates semantic factors (modifier type) and pragmatic factors (visual context), enabling disentangled analysis. Results show that both models exhibit strong sensitivity to modifier type—superlatives most robustly elicit ignorance inferences—but display weak, inconsistent responses to visual context, markedly underperforming humans. This reveals a fundamental limitation in VLMs’ visually grounded pragmatic reasoning and underscores their deficient capacity for context-dependent semantic-pragmatic integration.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This study explored how Vision-Language Models (VLMs) process ignorance implicatures with visual and linguistic cues. Particularly, we focused on the effects of contexts (precise and approximate contexts) and modifier types (bare numerals, superlative, and comparative modifiers), which were considered pragmatic and semantic factors respectively. Methodologically, we conducted a truth-value judgment task in visually grounded settings using GPT-4o and Gemini 1.5 Pro. The results indicate that while both models exhibited sensitivity to linguistic cues (modifier), they failed to process ignorance implicatures with visual cues (context) as humans do. Specifically, the influence of context was weaker and inconsistent across models, indicating challenges in pragmatic reasoning for VLMs. On the other hand, superlative modifiers were more strongly associated with ignorance implicatures as compared to comparative modifiers, supporting the semantic view. These findings highlight the need for further advancements in VLMs to process language-vision information in a context-dependent way to achieve human-like pragmatic inference.

Problem

• VLMs process ignorance implicatures
• Visual and linguistic cues effects
• Challenges in pragmatic reasoning

Innovation

• Utilized GPT-4o and Gemini 1.5 Pro
• Focused on linguistic and visual cues
• Conducted truth-value judgment tasks