Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

2026-08-28

AI Summary

论文针对大型视觉-语言模型产生幻觉的问题，提出了一种无需训练的推理时方法——动态对齐补偿（DAC），通过检测并修正跨模态表示的退化来减少幻觉。

Abstract

Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose \emph{Dynamic Alignment Compensation} (DAC), a training-free inference-time method that detects representation divergence and selectively applies lightweight residual compensation. DAC combines Layer-wise Semantic Compensation to mitigate inter-layer degradation with Sequential Semantic Correction to constrain temporal drift. Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.

Problem

• Large Vision-Language Models
• hallucinations
• internal representation dynamics
• autoregressive generation

Innovation

• Dynamic Alignment Compensation
• Layer-wise Semantic Compensation
• Sequential Semantic Correction