Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant

2026-01-24

AI Summary

This work proposes Memento, a wearable proactive augmented reality (AR) memory aid system designed to construct temporally coherent personal memory sequences by continuously capturing users’ spoken queries along with rich multimodal contextual data—including spatiotemporal coordinates, visual surroundings, physiological states, and ongoing activities. When encountering contexts similar to past experiences, Memento proactively delivers relevant and updated information through AR overlays, seamlessly integrating personalized assistance into everyday life. This study is the first to model user interactions as a coherent memory stream and introduces an end-to-end architecture that unifies speech recognition, context awareness, and spatiotemporal activity modeling. Preliminary user evaluations in real-world settings demonstrate the system’s effectiveness and yield key insights into the design challenges and practical considerations for proactive AR assistants.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Multiagent Systems: Applications

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We introduce Memento, a conversational AR assistant that permanently captures and memorizes user's verbal queries alongside their spatiotemporal and activity contexts. By storing these"memories,"Memento discovers connections between users'recurring interests and the contexts that trigger them. Upon detection of similar or identical spatiotemporal activity, Memento proactively recalls user interests and delivers up-to-date responses through AR, seamlessly integrating AR experience into their daily routine. Unlike prior work, each interaction in Memento is not a transient event, but a connected series of interactions with coherent long--term perspective, tailored to the user's broader multimodal (visual, spatial, temporal, and embodied) context. We conduct a preliminary evaluation through user feedbacks with participants of diverse expertise in immersive apps, and explore the value of proactive context-aware AR assistant in everyday settings. We share our findings and challenges in designing a proactive, context-aware AR system.

Problem

• proactive AR
• everyday memories
• context-aware assistant
• spatiotemporal context
• wearable AR

Innovation

• proactive AR
• context-aware assistant
• memory-based interaction
• wearable AR
• multimodal context