Can Hallucination Correction Improve Video-Language Alignment?

2025-02-20

AI Summary

Large vision-language models (VLMs) frequently generate hallucinated descriptions inconsistent with visual content, undermining cross-modal alignment in video-language tasks. To address this, we propose HACA, a self-training framework that explicitly models hallucination correction as an alignment learning signal—rather than merely suppressing hallucinations. HACA introduces a video-text inconsistency detection and reconstruction module to identify and rectify descriptive deviations, enabling self-driven consistency optimization without additional annotations. The method integrates contrastive learning with cross-modal attention to strengthen fine-grained spatiotemporal alignment. Evaluated on MSR-VTT and YouCook2 benchmarks, HACA achieves significant improvements in video-caption binding and text-to-video retrieval, yielding an average +3.2% gain in R@1. These results validate the effectiveness and generalizability of leveraging hallucination correction as a principled driver for cross-modal alignment.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model's ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.

Problem

• Improve video-language alignment
• Correct hallucinated content
• Enhance spatio-temporal reasoning

Innovation

• Hallucination correction training objective
• Self-training framework HACA
• Enhances video-text alignment