DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

2026-06-22

AI Summary

Current evaluations of large language models’ memory capabilities are largely confined to short-term, simplified interactions, failing to capture the long-term, heterogeneous, implicit, and externally driven nature of real-world user preference evolution. To address this gap, this work proposes DynamicMem—the first synthetic memory benchmark that jointly incorporates longitudinality, heterogeneous evolution, and context-driven dynamics. It simulates 15-month multi-application behavioral trajectories for each user across 16 domains (averaging 2.2M tokens and 1,772 grounded events), constructs dynamic user profiles from implicit signals, and evaluates memory systems at five quarterly checkpoints. Experiments reveal a significant decline in profile reconstruction fidelity with increasing history length, while task accuracy remains stable. All evaluated systems fail to simultaneously retain static facts and incorporate evolving information, with over 93% of failures attributable to memory retrieval errors—highlighting retrieval modules, not generative models, as the primary performance bottleneck.

Abstract

LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/

Problem

• long-horizon memory
• user profile evolution
• implicit memory signals
• real-world behavior
• memory benchmark

Innovation

• long-horizon memory
• synthetic benchmark
• multi-app inference
• user profile evolution
• memory retrieval