MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

2026-06-15

AI Summary

Current evaluations of long-term memory in large language models predominantly rely on accuracy metrics for isolated questions, which fail to capture the dynamic behavior of user-specific facts under varying conditions. This work proposes MemTrace, a novel benchmark that shifts the evaluation unit from individual questions to knowledge points and constructs multidimensional, controllable probes along three axes: memory age, question type, and evidence condition. Using this framework, we systematically evaluate 13 system configurations across four memory paradigms and reveal that high overall accuracy masks significant deficiencies in tracking factual evolution and correcting erroneous premises. These failures primarily stem not from missing retrieval but from an inability to effectively leverage available evidential context.

Abstract

LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change. We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings. Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.

Problem

• long-term memory
• knowledge point
• evidence use
• memory evaluation
• fact tracking

Innovation

• MemTrace
• long-term memory
• knowledge point
• evidence utilization
• memory benchmark