NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

2026-02-25

AI Summary

This work addresses the prevalent issue of object hallucination in large vision-language models (LVLMs), where generated descriptions include objects absent from the input image—a phenomenon primarily driven by strong language priors in the decoder. The study is the first to explicitly identify language priors as the dominant cause of such hallucinations and introduces NoLan, a training-free, plug-and-play framework that dynamically suppresses these priors during inference. NoLan adaptively modulates the influence of language priors by comparing the output distributions derived from multimodal inputs against those from text-only inputs. Without compromising other model capabilities, this approach significantly reduces hallucination rates on the POPE benchmark, improving accuracy by 6.45 and 7.21 points for LLaVA-1.5 7B and Qwen-VL 7B, respectively.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.

Problem

• object hallucination
• Large Vision-Language Models
• language priors
• vision-language alignment
• multimodal generation

Innovation

• object hallucination
• vision-language models
• language priors
• dynamic suppression
• training-free decoding