MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations

2026-07-14

AI Summary

Existing evaluations of long-context dialogue memory rely solely on final question-answering accuracy, making it difficult to pinpoint specific failure modes such as omissions, incorrect bindings, or reliance on outdated information. This work addresses this limitation by modeling dialogue memory as an explicit lifecycle comprising operations like remembering, forgetting, updating, and reflecting. The authors introduce a structured representation of memory events and six operation-level probing tasks, leveraging a controlled generation pipeline to construct the first long-dialogue benchmark with fine-grained, operation-level annotations. The framework supports both adjacent-evidence and long-context evaluation settings, revealing significant shortcomings in current models: for instance, turn-level retrieval underperforms session-level retrieval, and long-context models struggle to reconstruct temporally coherent memory state trajectories. This enables fine-grained, interpretable diagnosis of memory capabilities.

Abstract

Long-term memory has become a foundational capability for LLM-based agents that accompany users across extended, multi-session interactions. Existing benchmarks, however, evaluate such memory almost exclusively through downstream question answering, scoring only the correctness of a final answer. This black-box formulation conflates the heterogeneous causes of memory failure, such as missing the introduction of a relevant fact, binding an operation to the wrong target, or relying on stale values after a correction. As a result, it can credit correct answers despite their reliance on inconsistent or unsafe memory states. In this paper, we argue that, in dynamic long-horizon interactions, memory is not a static collection of facts but a lifecycle of explicit operations, including remembering, forgetting, updating, reflecting, and their compositions. We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations and represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence. A controllable generation pipeline embeds these operations into long, task-oriented conversations and produces gold operation traces together with six categories of operation-level probes, evaluated under both adjacent-evidence and long-context settings. Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals, revealing that current systems remain far from uniformly reliable. For instance, session-level retrieval outperforms turn-level retrieval, and long-context models remain notably weak at reconstructing ordered memory-state trajectories. These results move long-term memory evaluation from final-answer scoring toward interpretable, operation-level diagnosis.

Problem

• long-term memory
• memory operations
• benchmarking
• LLM-based agents
• memory failure

Innovation

• memory lifecycle operations
• structured memory trace
• operation-level evaluation
• long-horizon conversation
• controllable benchmark generation