Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

2026-06-13

AI Summary

This work proposes the first user-centric continual spatial intelligence task, aiming to address long-term spatial reasoning from streaming first-person videos in dynamic environments. To this end, the authors introduce the UCS-Bench benchmark dataset and present the DirectMe framework, which unifies visual perception, memory updating, and spatial reasoning through incrementally structured spatial memory modeling, multimodal large language model augmentation, and relative position tracking. The proposed approach significantly enhances the spatial localization and recall capabilities of state-of-the-art multimodal large models under viewpoint changes and dynamic scene conditions, outperforming existing spatial perception and long video stream models on UCS-Bench.

Abstract

We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.

Problem

• egocentric video
• spatial reasoning
• long-term memory
• user-centric intelligence
• continual reasoning

Innovation

• egocentric video
• spatial reasoning
• continual memory
• structured spatial memory
• user-centric AI