Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering

2026-01-31

AI Summary

This work addresses the pervasive hallucination problem in large vision-language models (LVLMs), which existing approaches often mitigate only at the output layer without probing the underlying representational mechanisms. For the first time, this study investigates hallucination origins at the neuronal level by applying sparse autoencoders to decompose visual embeddings, thereby identifying interpretable neurons correlated with actual image content. Building on this insight, the authors propose a contrastive neuron modulation strategy that, during the prefilling stage, enhances activation of content-relevant neurons while suppressing irrelevant ones to reduce hallucinatory outputs. Evaluated across multiple hallucination-specific and general multimodal benchmarks, the method significantly lowers hallucination rates while preserving or even improving the model’s visual-semantic comprehension capabilities, offering an interpretable, controllable, and effective solution to hallucination mitigation.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

LVLMs achieve remarkable multimodal understanding and generation but remain susceptible to hallucinations. Existing mitigation methods predominantly focus on output-level adjustments, leaving the internal mechanisms that give rise to these hallucinations largely unexplored. To gain a deeper understanding, we adopt a representation-level perspective by introducing sparse autoencoders (SAEs) to decompose dense visual embeddings into sparse, interpretable neurons. Through neuron-level analysis, we identify distinct neuron types, including always-on neurons and image-specific neurons. Our findings reveal that hallucinations often result from disruptions or spurious activations of image-specific neurons, while always-on neurons remain largely stable. Moreover, selectively enhancing or suppressing image-specific neurons enables controllable intervention in LVLM outputs, improving visual grounding and reducing hallucinations. Building on these insights, we propose Contrastive Neuron Steering (CNS), which identifies image-specific neurons via contrastive analysis between clean and noisy inputs. CNS selectively amplifies informative neurons while suppressing perturbation-induced activations, producing more robust and semantically grounded visual representations. This not only enhances visual understanding but also effectively mitigates hallucinations. By operating at the prefilling stage, CNS is fully compatible with existing decoding-stage methods. Extensive experiments on both hallucination-focused and general multimodal benchmarks demonstrate that CNS consistently reduces hallucinations while preserving overall multimodal understanding.

Problem

• hallucination
• large vision-language models
• interpretable neurons
• visual grounding
• multimodal understanding

Innovation

• Contrastive Neuron Steering
• sparse autoencoders
• interpretable neurons
• hallucination mitigation
• visual grounding