Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

2026-08-07

AI Summary

This work addresses the challenge that existing multimodal large language models struggle to efficiently process ultra-long videos spanning several hours to days, and that conventional approaches incur excessive complexity when constructing memory for unseen queries. To overcome these limitations, the authors propose MERIT, a two-stage framework that first builds a lightweight, query-agnostic memory and then leverages retrieval augmentation during inference. Key innovations include a multi-key episodic memory representation, a high-recall retrieval mechanism based on key matching, and a dynamic neighborhood filtering strategy that expands local temporal context at inference time while avoiding the computational burden of global modeling. The method achieves state-of-the-art performance across three ultra-long video benchmarks: EgoLifeQA, LVBench, and Video-MME (Long).

Abstract

When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs). This ultra-long setting necessitates a two-stage paradigm: query-agnostic memory construction followed by retrieval-based inference. Prior work invests in complex memory construction to pre-model high-level relations in videos, despite not knowing the downstream query at build time. We instead prioritize high-recall retrievability during memory building, and defer query-specific, high-level relation composition to inference time. To this end, we propose MERIT(Multi-key Episodic Retrieval with Inference-time Temporal expansion), a simple yet effective agentic framework for ultra-long video understanding. First, we formulate an episodic multi-key representation that enables precise retrieval of fine-grained memories through a simple key-matching mechanism. Second, we introduce a neighbor filtering mechanism to capture broader semantic context without the massive computational overhead of global memory construction. This is achieved by expanding the temporal scope exclusively around the retrieved segments at inference time. By leveraging simple key-matching with this on-demand temporal expansion, MERIT achieves state-of-the-art performance across three long-video benchmarks: EgoLifeQA, LVBench, and Video-MME (Long).

Problem

• ultra-long video understanding
• episodic memory retrieval
• multi-key representation
• retrieval-based inference
• memory construction

Innovation

• multi-key retrieval
• episodic memory
• inference-time expansion
• ultra-long video understanding
• retrieval-based reasoning