MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery

2026-06-23

AI Summary

This work addresses the lack of direct evaluation of long-term memory content in large language model agents, which currently rely solely on downstream behavioral proxies that hinder auditability of retained user states. The authors propose treating long-term memory as an auditable artifact by directly assessing its quality through reconstruction of latent, structured user states. To this end, they introduce MEMPROBE—the first benchmark for memory recovery capability—featuring a synthetic ground-truth repository of hidden user states, simulated user trajectories, controlled information leakage tasks, and balanced state dimensions. Evaluations under both full-storage and top-k retrieval settings reveal a significant gap between task completion performance and memory fidelity: despite high task success rates across 50 users and 1,550 targets, memory recovery rates hover around 0.6 and further decline under top-k retrieval, underscoring a critical disconnect between functional assistance and faithful memory retention.

Abstract

Long-term memory promises LLM agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that interaction forms. In practice, however, this memory is evaluated mostly through downstream behavior, such as later answers, personalization quality, or task success, which tests that understanding only indirectly and leaves the memory artifact itself largely unaudited. We argue that long-term memory should instead be evaluated as an auditable post-interaction artifact: after ordinary assistance, what structured user state can be reconstructed from the memory the agent leaves behind? We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users, each carrying a hidden, taxonomy-anchored user-state bank, across a trajectory of leak-controlled tasks, after which that bank is reconstructed from the agent's resulting memory under both full-store and top-k access. Built on synthetic ground truth for efficient, scalable measurement, MEMPROBE spans 50 simulated users with 31 hidden dimensions each (1,550 recovery targets) and tests 5 representative memory systems. Testing state-of-the-art memory agents, we find that successful assistance and recoverable memory behave as distinct capabilities. Task completion nearly saturates, even for a memoryless baseline, while category-balanced recovery stays moderate (about 0.6) and drops further under top-k retrieval. MEMPROBE is the first benchmark to study memory recovery directly, reconstructing the user state a system retains and scoring it against ground truth. We see recovery as a concrete objective for future memory agents to optimize, and MEMPROBE as a step toward an environment where agents are trained to remember their users, growing more faithful the longer they know them.

Problem

• long-term memory
• user-state recovery
• memory evaluation
• LLM agents
• auditable memory

Innovation

• long-term memory
• memory recovery
• user-state reconstruction
• auditable memory
• benchmarking