S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval

2026-07-02

AI Summary

Existing memory retrieval benchmarks predominantly rely on offline evaluation, failing to capture the dynamics of real-world recall scenarios under continuous video streams from wearable devices. This work introduces and formalizes the "streaming episodic memory retrieval" paradigm, presenting the first large-scale benchmark built from authentic first-person videos captured by Ray-Ban Meta smart glasses. The dataset comprises 388 hours of video paired with 9,448 temporally precise, variable-length question-answer pairs, enabling event-triggered, causality-driven real-time memory access. Experiments uncover a “localization paradox” in large language models: while semantic reasoning improves with model scale, temporal localization accuracy remains fundamentally constrained by architectural limitations, showing minimal gains from further scaling or increased frame sampling density.

Abstract

As wearable devices enable continuous first-person recording, AI assistants must reason across long time horizons to recall past experiences-a capability known as episodic memory. Current benchmarks often rely on offline evaluation with access to entire video files, failing to simulate the streaming reality of wearable intelligence. We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses. S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream. We provide 9,448 QA pairs requiring manual visual proof through precise temporal localization and supporting flexible response lengths to simulate natural human-AI interaction. Our extensive benchmarking of frontier models uncovers a localization paradox: while semantic reasoning improves with parameter scale, temporal grounding precision remains a stagnant architectural bottleneck that does not benefit from brute-force increases in model size, resolution, or frame density. S-EMBER establishes a hardware-authentic foundation for developing grounded, reliable episodic memory in the next generation of wearable AI agents.

Problem

• streaming egocentric memory
• episodic retrieval
• wearable AI
• temporal grounding
• first-person video

Innovation

• streaming episodic memory
• egocentric video
• temporal grounding
• wearable AI
• causal retrieval