LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

2026-04-20

AI Summary

This work addresses the lack of systematic evaluation of hallucination behaviors in current vision-language models (VLMs) under prompts with varying levels of linguistic assertiveness, particularly the fine-grained impact of verbal pressure on generating unfounded responses. We propose Ghost-100, a benchmark comprising 800 synthetic images paired with five-tiered assertiveness levels in prompts. By fixing both image content and task while only varying instruction tone, we isolate assertiveness as an independent variable to assess model hallucinations in fact-negating scenarios. Our framework introduces the first structured assertiveness gradient combined with negative sample design, along with dual metrics—H-Rate and LLM-based H-Score. Experiments across nine open-source VLMs reveal a significant decoupling between H-Rate and H-Score, task-dependent response patterns to assertiveness, and a non-monotonic sensitivity wherein hallucinations peak at moderate assertiveness levels.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavior under progressively coercive prompt phrasing remains undercharacterized. Existing hallucination benchmarks predominantly rely on neutral prompts and binary detection, leaving open how both the incidence and the intensity of fabrication respond to graded linguistic pressure across structurally distinct task types. We present Ghost-100, a procedurally constructed benchmark of 800 synthetically generated images spanning eight categories across three task families -- text-illegibility, time-reading, and object-absence -- each designed under a negative-ground-truth principle that guarantees the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework, holding the image and task identity fixed while varying only directive force, so that tone is isolated as the sole independent variable. We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1-5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow, which retrospectively confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate substantially across model families, reading-style and presence-detection subsets respond to prompt pressure in qualitatively different ways, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels -- patterns that aggregate metrics obscure.

Problem

• hallucination
• vision-language models
• prompt intensity
• tone-induced
• visual grounding

Innovation

• tone-induced hallucination
• prompt intensity framework
• negative-ground-truth benchmark
• dual-track evaluation
• vision-language models