Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment

2026-04-13

AI Summary

This work addresses the challenges of memory bloat and inefficient real-time querying in long-term robot deployment, where continuous multimodal perception data overwhelms storage and impedes retention of user-relevant experiences. To tackle this, the authors propose a hierarchical episodic memory system that uniquely integrates user feedback–driven natural language rules with a large language model to assess memory relevance and enable selective forgetting, while supporting dynamic updates of memory policies. Evaluated on simulated household tasks and 20.5 hours of real-world robot data, the approach reduces memory footprint by 45%, lowers query computational overhead by 35%, and improves second-round question-answering accuracy by 70%, significantly enhancing the scalability and personalization of long-term robotic memory.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Machine Learning: Life-Long and Continual Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Robots must verbalize their past experiences when users ask "Where did you put my keys?" or "Why did the task fail?" Yet maintaining life-long episodic memory (EM) from continuous multimodal perception quickly exceeds storage limits and makes real-time query impractical, calling for selective forgetting that adapts to users' notions of relevance. We present H$^2$-EMV, a framework enabling humanoids to learn what to remember through user interaction. Our approach incrementally constructs hierarchical EM, selectively forgets using language-model-based relevance estimation conditioned on learned natural-language rules, and updates these rules given user feedback about forgotten details. Evaluations on simulated household tasks and 20.5-hour-long real-world recordings from ARMAR-7 demonstrate that H$^2$-EMV maintains question-answering accuracy while reducing memory size by 45% and query-time compute by 35%. Critically, performance improves over time - accuracy increases 70% in second-round queries by adapting to user-specific priorities - demonstrating that learned forgetting enables scalable, personalized EM for long-term human-robot collaboration.

Problem

• lifelong episodic memory
• selective forgetting
• human-robot interaction
• memory scalability
• personalized memory

Innovation

• hierarchical episodic memory
• selective forgetting
• language-model-based relevance
• user-guided learning
• lifelong robot memory