ReflectWorld-MM: An Entity-Oriented Multi-Media Memory System for Open-Ended Video Streams

2026-07-06

AI Summary

Current video understanding systems are constrained by frame-level flat memory structures, which struggle to track and retain persistent entities over open-ended video streams. This work proposes an entity-centric multimodal memory system that parses video streams into structured entity observations and introduces a hierarchical long-term memory architecture inspired by human memory theories. The architecture integrates multiscale episodic memory, evolving semantic memory, and procedural memory. By coupling a perception frontend with off-the-shelf agents, the system enables deployment in real-world scenarios. Evaluated across six long-video and lifelong memory benchmarks, the proposed method achieves state-of-the-art accuracy, significantly outperforming strong memory-based agent baselines and leading large models.

Abstract

Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-standing goal, and recently multimodal agents equipped with long-term memory over video streams have attracted increasing interest. Unfortunately, existing systems either keep their memory inside the model context or in a flat feature store, and organize it around frames rather than around the persistent entities a stream is really about, which confines them to bounded videos and weakens their ability to track who and what reappears over time. In this paper, we propose ReflectWorld-MM, an entity-oriented multi-media memory system for open-ended video streams. It consists of three parts. The first is a perception front-end that turns a streaming video into entity-resolved observations under a bounded short-term memory. The second is a hierarchical long-term memory, grounded in human memory theory, that couples a multi-scale episodic memory, an evolving entity-centric semantic memory, and a procedural memory. The third is a complete realization, built for real-world operation, that ingests arbitrary streams and plugs into off-the-shelf assistants. Across six long-video and lifelong-memory benchmarks, ReflectWorld-MM achieves the best accuracy on all six, outperforming strong memory agents and a frontier model.

Problem

• open-ended video streams
• long-term memory
• entity-oriented memory
• multimodal agents
• video understanding

Innovation

• entity-oriented memory
• multi-media memory system
• open-ended video streams
• hierarchical long-term memory
• lifelong memory