Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

2026-08-10

AI Summary

This work addresses the prevalent issue of hallucination in large vision-language models (LVLMs), where generated text often contradicts visual input, undermining model reliability. The authors propose a training-free, test-time hallucination suppression framework that dynamically constructs a disentangled hallucination subspace for each input instance and selectively suppresses the most likely hallucinatory directions via adaptive weighted projection, while preserving image-relevant semantics. Moving beyond conventional global editing strategies, this approach achieves instance-level dynamic modeling of hallucinations for the first time, substantially improving both suppression accuracy and generalization. Extensive experiments demonstrate consistent performance gains across diverse LVLM architectures and vision-language benchmarks, confirming the framework’s robustness, versatility, and efficiency.

Abstract

Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.

Problem

• hallucination
• large vision-language models
• model editing
• disentangled subspaces
• training-free

Innovation

• disentangled subspaces
• training-free
• hallucination mitigation
• per-instance editing
• LVLMs