🤖 AI Summary
This work addresses the common misattribution of object hallucination in multimodal large language models to visual neglect. The authors propose a training-free, inference-time intervention framework that finely modulates the model’s reliance on visual context versus parametric knowledge through a contextual preference vector and a single-step residual injection mechanism applied at intermediate MLP layers. By integrating Contextual Preference Activation Steering (CAS) with carefully designed conflict samples, the method effectively mitigates object hallucinations without increasing decoding latency or compromising text generation quality, achieving a significant reduction in hallucination rates.
📝 Abstract
Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.