Test-Time Hallucination Control in Large Vision-Language Models

2026-08-11

AI Summary

This work addresses the prevalent issue of object hallucination in large vision-language models, where generated image captions often contain factually incorrect content inconsistent with the input image, thereby undermining model reliability. The authors propose a training-free, test-time hallucination mitigation method (TTH) that leverages a zero-shot multimodal classifier to produce image-grounded auxiliary logits for candidate object tokens. These logits are fused at the token level with the model’s original outputs, guided by an entropy-based weighting strategy to refine predictions. Notably, this approach achieves effective and general-purpose hallucination control without requiring multiple decoding passes or architectural modifications to the underlying model. Extensive experiments demonstrate significant improvements in accuracy and robustness across diverse vision-language models and benchmarks, while fully preserving the models’ pretrained knowledge.

Abstract

Object Hallucination in large vision-language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the token level for object tokens selected from a candidate pool. An entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH

Problem

• Object Hallucination
• Large Vision-Language Models
• Test-Time Control
• Reliability
• Non-factual Generation

Innovation

• Test-Time Hallucination Mitigation
• Zero-Shot Multi-Modal Classifier
• Token-Level Fusion
• Entropy-Based Weighting
• Training-Free Method