SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

2026-07-05

AI Summary

This work addresses the issue of image-inconsistent hallucinations in large vision-language models (LVLMs), which often arise from irrelevant or noisy visual tokens. While existing approaches are largely confined to the decoding stage, this paper proposes SeeMe—a training-free, three-stage framework for visual token reconstruction that pioneers visual token engineering as a core strategy for hallucination mitigation. By selectively refining visual tokens through selection, reweighting, and recombination—guided by attention analysis—SeeMe effectively purifies the input visual representation, thereby suppressing misleading signals at their source. Evaluated on three major benchmarks (MME, POPE, and AMBER), SeeMe substantially reduces hallucination rates across four prominent LVLMs while simultaneously enhancing the consistency between model outputs and the input images.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Problem

• hallucinations
• Large Vision-Language Models
• visual tokens
• visual understanding
• token engineering

Innovation

• visual token engineering
• hallucination mitigation
• training-free framework
• large vision-language models
• feature engineering