Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning

2026-05-07

AI Summary

Existing approaches struggle to effectively comprehend ultra-long videos spanning days to weeks, hindered by sparse frame sampling, fragmented multimodal information, and insufficient modeling of long-term narratives. To address these limitations, this work proposes MAGIC-Video, a novel framework that uniquely integrates a structured multimodal memory graph—featuring six edge types—with long-range narrative chains. Through an agent-based reasoning loop, the framework dynamically fuses graph retrieval with injected narrative facts, enabling training-free, unified cross-modal and cross-temporal understanding. Evaluated on EgoLifeQA, Ego-R1, and MM-Lifelong benchmarks, MAGIC-Video outperforms previous state-of-the-art agent systems by 10.1, 7.4, and 5.9 percentage points, respectively, substantially overcoming the fragmentation inherent in current systems across both temporal and modal dimensions.

Abstract

Understanding ultra-long videos such as egocentric recordings, live streams, or surveillance footage spanning days to weeks, remains a challenge. For current multimodal LLMs: even with million-token context windows, frame budgets cover only tens of minutes of densely sampled video, and most evidence is discarded before inference begins. Memory-augmented and agentic approaches help with scale, but their retrieval remains fragmented across modalities and lacks long-range narrative summaries that span days or weeks. We propose \textbf{MAGIC-Video}, a training-free framework built around a multimodal memory graph with interleaved narrative chain: the graph unifies episodic, semantic, and visual content through six typed edges and supports cross-modal retrieval, while the chain distils long-horizon entity biographies and recurring activity events. At inference time, an agentic loop interleaves graph retrieval with narrative fact injection, covering both the modality and time dimensions of ultra-long video in a single retrieval pipeline. On EgoLifeQA, Ego-R1 and MM-Lifelong, MAGIC-Video consistently outperforms strong general-purpose, long-video, and agentic baselines, with gains of 10.1, 7.4, and 5.9 points over the prior best agentic system on each benchmark. Code is available at https://github.com/lijiazheng0917/MAGIC-video.

Problem

• ultra-long video reasoning
• multimodal retrieval
• long-range narrative
• memory-augmented systems
• agentic video understanding

Innovation

• multimodal memory graph
• narrative chain
• ultra-long video reasoning
• agentic retrieval
• cross-modal integration