StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall

2026-04-28

AI Summary

This work addresses a critical limitation in existing dialogue system benchmarks, which treat memory as a static repository of facts and overlook the need for virtual characters to strategically access and deploy memories to balance factual accuracy with social objectives during conversation. To bridge this gap, we introduce StratMem-Bench, the first evaluation benchmark specifically designed to assess strategic memory usage in character-driven dialogue. It comprises 657 dialogue instances paired with heterogeneous memory pools containing essential, supportive, and irrelevant memory entries. We further propose multidimensional automatic metrics, including Strict Memory Compliance and Memory Integration Quality, to holistically evaluate model performance. Experimental results reveal that while state-of-the-art large language models effectively retrieve essential memories, their ability to coherently integrate supportive memories deteriorates significantly, exposing fundamental limitations in executing nuanced memory strategies.

Abstract

Achieving realistic human-like conversation for virtual characters requires not only a simple memorization and recall of past events, but also the strategic utilization of memory to meet factual needs and social engagement. Current memory utilization relevant (e.g., memory-augmented generation, long-term dialogue, and etc.) benchmarks overlook this nuance, treating memory primarily as a static repository of facts rather than a dynamic resource to be strategically deployed in dialogues. To address this gap, we design StratMem-Bench, a new benchmark to evaluate strategic memory use in character-centric dialogues. This dataset comprises 657 instances where virtual characters must navigate heterogeneous memory pools containing required, supportive, and irrelevant memories. We also propose a framework with different evaluation metrics including Strict Memory Compliance, Memory Integration Quality, Proactive Enrichment Score and Conditional Irrelevance Rate, to evaluate strategic memory use capabilities of virtual characters. Experiments on StratMem-Bench which leverage the state-of-the-art large language models as virtual characters show that all models perform well at distinguishing between required and irrelevant memories, but struggle once supportive memories are introduced into the decision process.

Problem

• strategic memory
• virtual character
• conversation
• memory utilization
• dialogue benchmark

Innovation

• strategic memory use
• virtual character conversation
• memory-augmented generation
• dialogue benchmark
• heterogeneous memory pool