Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations

2025-09-14

AI Summary

Large Vision-Language Models (LVLMs) suffer from pervasive visual-textual inconsistency hallucinations. Existing mitigation approaches rely on costly human annotations or auxiliary models to construct preference data, limiting scalability and practicality. This paper proposes APASI—a self-injected, supervision-free hallucination mitigation framework. Its core innovation lies in leveraging the LVLM itself to generate high-quality negative samples that conform to authentic hallucination patterns, thereby constructing unsupervised preference pairs; it further employs iterative curriculum-aligned training to progressively enhance the model’s discriminative and generative capabilities for visual consistency. Evaluated across six benchmarks, APASI significantly reduces hallucination rates across three representative LVLMs, achieving performance on par with—or even surpassing—that of supervised alignment methods. These results demonstrate APASI’s effectiveness, generalizability, and potential for sustainable optimization.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large Vision-Language Models (LVLMs) suffer from serious hallucination problems, where the model-generated responses are inconsistent with the visual inputs. Existing hallucination mitigation methods are mainly based on preference alignment and require external human annotations or auxiliary models for preference data collection, which increase costs and limit sustainable improvement. To tackle these challenges, we propose Autonomous Preference Alignment via Self-Injection (APASI), a novel and generalizable method that mitigates hallucinations without external dependencies. APASI leverages the target LVLM to self-inject hallucinations into a generated response, creating a pair of responses with varying preference levels. During the self-injection process, the dis-preferred response is generated based on three key observations of hallucinations, ensuring it simulates real hallucination patterns. This fidelity offers an accurate learning signal for hallucination mitigation. Moreover, APASI incorporates an iterative alignment training strategy combined with curriculum learning to periodically update the preference data with increasing challenge, enabling stable and continuous enhancement of the LVLM. Extensive experiments across six benchmarks show that APASI not only effectively mitigates hallucinations for three baseline models but also achieves comparable or even superior performance to alignment-based methods with external dependency, thereby demonstrating its effectiveness and generalization capability. The code is available at https://github.com/davidluciolu/APASI.

Problem

• Mitigating hallucinations in vision-language models
• Reducing dependency on external annotation resources
• Self-generating preference data for alignment training

Innovation

• Self-injecting hallucinations for autonomous preference alignment
• Iterative alignment training with curriculum learning strategy
• Generating dis-preferred responses using key hallucination observations