SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decoding

2026-02-10

AI Summary

This work addresses the hallucination problem in large vision-language models (LVLMs), which undermines the reliability of generated content. It is the first to systematically uncover the internal patterns of hallucinations from three perspectives: attention heads, model layers, and decoding tokens. The authors propose a training-free dynamic decoding method that leverages an inter-layer Knowledge Stability Score (KSS) to compare knowledge representations between the most stable and least stable layers, thereby dynamically guiding the generation process. This plug-and-play approach is compatible with various mainstream architectures and achieves state-of-the-art hallucination suppression across multiple models, tasks, and benchmarks, significantly enhancing output faithfulness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Hallucinations in Large Vision-Language Models (LVLMs) pose significant security and reliability risks in real-world applications. Inspired by the observation that humans are more error-prone when uncertain or hesitant, we investigate how instability in a model's internal knowledge contributes to LVLM hallucinations. We conduct extensive empirical analyses from three perspectives, namely attention heads, model layers, and decoding tokens, and identify three key hallucination patterns: (i) visual activation drift across attention heads, (ii) pronounced knowledge fluctuations across layers, and (iii) visual focus distraction between neighboring output tokens. Building on these findings, we propose Stability-Aware Knowledge-Enhanced Decoding (SAKED), which introduces a layer-wise Knowledge Stability Score (KSS) to quantify knowledge stability throughout the model. By contrasting the most stability-aware and stability-agnostic layers, SAKED suppresses decoding noise and dynamically leverages the most reliable internal knowledge for faithful token generation. Moreover, SAKED is training-free and can be seamlessly integrated into different architectures. Extensive experiments demonstrate that SAKED achieves state-of-the-art performance for hallucination mitigation on various models, tasks, and benchmarks.

Problem

• hallucination
• Large Vision-Language Models
• reliability
• security
• knowledge stability

Innovation

• hallucination mitigation
• knowledge stability
• vision-language models
• decoding strategy
• training-free method