VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Models

2026-01-15

AI Summary

This work addresses the susceptibility of video large language models to hallucinations in reasoning about event relationships—such as causality, temporal order, and sub-event structures—a problem that has lacked systematic investigation. We formalize and evaluate video event relationship hallucination for the first time, introducing VERHallu, the first dedicated benchmark encompassing relation classification, question answering, and counterfactual QA tasks, featuring counterintuitive scenarios and human-annotated bias labels. To mitigate these hallucinations, we propose a Keyframe Propagation (KFP) strategy that dynamically redistributes frame-level attention at intermediate layers, enhancing the model’s awareness of multi-event context without compromising inference speed. Experimental results demonstrate that our approach significantly alleviates event relationship hallucinations and improves accuracy in understanding complex events.

Technology Category

• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video Large Language Models (VideoLLMs) exhibit various types of hallucinations. Existing research has primarily focused on hallucinations involving the presence of events, objects, and scenes in videos, while largely neglecting event relation hallucination. In this paper, we introduce a novel benchmark for evaluating the Video Event Relation Hallucination, named VERHallu. This benchmark focuses on causal, temporal, and subevent relations between events, encompassing three types of tasks: relation classification, question answering, and counterfactual question answering, for a comprehensive evaluation of event relation hallucination. Additionally, it features counterintuitive video scenarios that deviate from typical pretraining distributions, with each sample accompanied by human-annotated candidates covering both vision-language and pure language biases. Our analysis reveals that current state-of-the-art VideoLLMs struggle with dense-event relation reasoning, often relying on prior knowledge due to insufficient use of frame-level cues. Although these models demonstrate strong grounding capabilities for key events, they often overlook the surrounding subevents, leading to an incomplete and inaccurate understanding of event relations. To tackle this, we propose a Key-Frame Propagating (KFP) strategy, which reallocates frame-level attention within intermediate layers to enhance multi-event understanding. Experiments show it effectively mitigates the event relation hallucination without affecting inference speed.

Problem

• event relation hallucination
• Video Large Language Models
• causal relations
• temporal relations
• subevent relations

Innovation

• event relation hallucination
• Video Large Language Models
• VERHallu benchmark
• Key-Frame Propagating
• counterfactual reasoning