NEST: Narrative Event Structures in Time for Long Video Understanding

2026-06-17

AI Summary

Existing approaches to long-form video understanding struggle to model high-order narrative structures, failing to effectively capture the hierarchical composition from actions to events, temporal interactions among events, and long-range dependencies across extended durations. This work systematically introduces narrative structure into video understanding by constructing a dataset of 1,005 full-length movies, each annotated with 102 multimodal narrative events. These events are structurally linked through temporal ordering, hierarchical composition, and long-range dependencies, enabling four core tasks: event trigger detection, localization, argument extraction, and relation extraction. Baseline methods based on multimodal fusion prove highly challenging for the first three tasks, achieving F1 scores below 11%, while event relation extraction yields 35.45% and 44.42% F1 under zero-shot and fine-tuned settings, respectively, thereby addressing a critical gap in narrative modeling for semantic understanding of long videos.

Abstract

Recent progress in vision-language models has enabled the processing of increasingly long video sequences, but the ability to handle extended token streams does not translate to understanding of narrative structure in long videos. Existing long video benchmarks focus on needle-in-a-haystack retrieval rather than evaluating how low-level actions form events, how events interact across time, and how narratives progress, for example, whether a model can connect an early setback, such as a job loss to a later relationship breakup, despite long gaps, intervening scenes, or flashbacks that reframe what occurred. We introduce NEST (Narrative Event Structures in Time for Long Video Understanding), a dataset of 1005 full-length movies (avg. 98 minutes), each annotated with 102 multimodal narrative events grounded in visual content, dialogue, and audio. NEST captures multimodal narrative events with structured annotations grounded in visual content, dialogue, and audio, and links them through relations that reflect narrative structure, including temporal ordering, hierarchical composition, and long-range dependencies. We introduce baselines for event trigger detection (ETD), event localization (EL), event argument extraction (EAE), and event relation extraction (ERE). The benchmark is highly challenging for grounded event discovery, with ETD below 8%, EL under 6%, and EAE below 11%. In contrast, ERE is more tractable once events are given, reaching 35.45% F1 zero-shot and 44.42% F1 after fine-tuning.

Problem

• narrative structure
• long video understanding
• event relations
• temporal dependencies
• multimodal events

Innovation

• narrative structure
• long video understanding
• multimodal event annotation
• event relation extraction
• temporal reasoning