Interactive Episodic Memory with User Feedback

2026-04-27

AI Summary

This work addresses the challenge of inaccurate predictions in natural language–driven episodic memory tasks, where user queries are often ambiguous or underspecified and existing approaches lack mechanisms for interactive refinement. To bridge this gap, the authors propose the EM-QnF framework, which introduces user feedback into episodic memory retrieval for the first time. The framework features a plug-and-play Feedback Alignment Module (FALM) and a lightweight training strategy that enable the model to iteratively refine answer localization through multi-turn interactions. Evaluated on long-duration, first-person videos paired with natural language queries, the system significantly outperforms current methods across three challenging benchmarks, achieving performance on par with or better than commercial large vision-language models and demonstrating strong real-world generalization when guided by human feedback.

Abstract

In episodic memory with natural language queries (EM-NLQ), a user may ask a question (e.g., "Where did I place the mug?") that requires searching a long egocentric video, captured from the user's perspective, to find the moment that answers it. However, queries can be ambiguous or incomplete, leading to incorrect responses. Current methods ignore this key aspect and address EM-NLQ in a one-shot setup, limiting their applicability in real-world scenarios. In this work, we address this gap and introduce the Episodic Memory with Questions and Feedback task (EM-QnF). Here, the user can provide feedback on the model's initial prediction or add more information (e.g., "Before this. I'm looking for the big blue mug not the white one"), helping the model refine its predictions interactively. To this end, we collect datasets for feedback-based interaction and propose a lightweight training scheme that avoids expensive sequential optimization. We also introduce a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively. Our approach significantly improves over the state of the art on three challenging benchmarks and is better than or competitive with commercial large vision-language models while remaining efficient. Evaluation with human-generated feedback shows that it generalizes well to real-world scenarios.

Problem

• Episodic Memory
• Natural Language Queries
• User Feedback
• Ambiguity
• Interactive Refinement

Innovation

• interactive episodic memory
• user feedback
• Feedback ALignment Module
• EM-QnF
• egocentric video understanding