MemConflict: Evaluating Long-Term Memory Systems Under Memory Conflicts

2026-05-20

AI Summary

Existing long-term memory systems lack effective evaluation of memory retrieval and ranking capabilities under conflicting scenarios, making it difficult to assess their reliability in terms of temporal validity, factual correctness, and contextual appropriateness. This work proposes MemConflict, a diagnostic framework that systematically defines and simulates three types of memory conflicts—dynamic, static, and conditional—and constructs a multi-turn dialogue benchmark incorporating cross-session conflicts and semantic interference. The framework enables both black-box answer evaluation and white-box memory analysis, revealing a notable disconnect between answer correctness and underlying memory quality. Experiments across six state-of-the-art systems demonstrate significant performance degradation under various conflict conditions, with pronounced sensitivity to conversation history length, distractor presence, implicit queries, and conflict distance.

Abstract

Long-term memory systems enable conversational agents based on large language models (LLMs) to retain, retrieve, and apply user-specific information across multi-session interactions. However, existing evaluations mainly assess outcome-level performance or temporal updating, providing limited insight into how systems retrieve and rank temporally valid, factually correct, and contextually applicable memory evidence under conflicting alternatives. To address this gap, we propose MemConflict, a diagnostic framework that treats memory validity as a query-conditioned fitness-for-use problem. MemConflict formalizes dynamic, static, and conditional conflicts over temporal validity, factual correctness, and contextual applicability. It simulates controlled long-horizon histories from structured user profiles, introduces cross-session conflicts, and injects semantically similar distractors to create competition among memory candidates. The resulting multi-session dialogue benchmark supports black-box evaluation of final answers and white-box analysis of supporting-memory retrieval and ranking. Experiments on six representative long-term memory systems show uneven strengths across conflict types, with answer correctness often diverging from memory retrieval and ranking. Sensitivity analyses reveal that longer histories, distractors, implicit queries, and larger conflict distances degrade performance. Diagnostics show failures from missing supporting memories and ineffective use of retrieved memories. Collectively, MemConflict advances principled long-term memory governance through retrieval-aware, conflict-aware reliability assessment.

Problem

• long-term memory
• memory conflict
• retrieval
• ranking
• evaluation

Innovation

• memory conflict
• long-term memory systems
• retrieval-aware evaluation
• diagnostic benchmark
• LLM memory governance