TPC: Cross-Temporal Prediction Connection for Vision-Language Model Hallucination Reduction

2025-03-06

AI Summary

Vision-language models (VLMs) are prone to hallucination in image captioning due to over-reliance on linguistic priors, compromising reliability in high-stakes applications. To address this, we propose a parameter-free, training-free temporal logits consistency modeling framework. Our method introduces Temporal Prediction Connection (TPC), the first mechanism to explicitly enforce semantic continuity across autoregressive decoding steps by linking logits temporally. It integrates temperature scaling with cross-step consistency regularization to stabilize generation without architectural modification. Evaluated across multiple benchmarks, our approach reduces hallucination rates by 23.6% on average, while improving caption accuracy and textual coherence. Crucially, it incurs no computational overhead—preserving inference speed—and maintains strong open-domain robustness. The method requires neither additional parameters nor fine-tuning, offering a lightweight, plug-and-play solution for hallucination mitigation in VLM-based captioning.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-language models (VLMs) have achieved remarkable advancements, capitalizing on the impressive capabilities of large language models (LLMs) across diverse tasks. Despite this, a critical challenge known as hallucination occurs when models overconfidently describe objects or attributes absent from the image, a problem exacerbated by the tendency of VLMs to rely on linguistic priors. This limitation reduces model reliability in high-stakes applications. In this work, we have observed the characteristic of logits' continuity consistency enhancement and introduced a straightforward and efficient method, Cross-Temporal Prediction Connection (TPC), designed to enhance the semantic consistency of logits by connecting them temporally across timesteps. TPC amplifies information flow and improves coherence, effectively reducing hallucination. Extensive experiments show that TPC surpasses existing representatives, delivering superior performance in both accuracy and efficiency while maintaining robustness in open-ended text generation tasks.

Problem

• Reduces hallucination in vision-language models
• Enhances semantic consistency across timesteps
• Improves reliability in high-stakes applications

Innovation

• Cross-Temporal Prediction Connection (TPC) introduced
• Enhances semantic consistency of logits temporally
• Reduces hallucination in vision-language models effectively