From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

2026-04-21

AI Summary

Current evaluations of long-term memory are largely confined to static fact retrieval, failing to assess an agent’s ability to integrate memories and adapt to evolving knowledge over extended interactions. This work introduces Memora, a benchmark comprising user dialogues spanning weeks to months, which systematically evaluates long-term memory through tasks in memorization, reasoning, and recommendation. We propose the Forgetting-Aware Memory Accuracy (FAMA) metric, which explicitly penalizes reliance on outdated or invalid memories. Combining automated memory anchoring verification, human evaluation, and FAMA, we benchmark four large language models and six memory-augmented agents, revealing that existing approaches frequently reuse obsolete information and struggle to reconcile dynamic updates, resulting in only marginal performance gains from current memory mechanisms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.

Problem

• long-term memory
• personalized agents
• memory forgetting
• knowledge updates
• memory evaluation

Innovation

• long-term memory
• personalized agents
• memory benchmarking
• forgetting-aware evaluation
• FAMA