Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents

2026-06-23

AI Summary

This study addresses the limited understanding of how the functional roles of conversational memory influence the response quality of large language models, a gap exacerbated by the inability of conventional evaluation methods to capture nuanced user preferences. To bridge this, the work introduces the first fine-grained functional taxonomy of conversational memory and develops a user-centered, reference-free evaluation framework. Empirical experiments are conducted using a retrieval-augmented generation (RAG) system and long-horizon dialogue data. Results demonstrate that clarifying-type memory significantly enhances factual accuracy and adherence to constraints in model responses, whereas irrelevant memory degrades topical relevance. These findings provide both theoretical grounding and empirical evidence for designing more personalized and controllable conversational agents.

Abstract

Prior research on memory mechanism in RAG-based conversational system has emphasized how memory is stored and retrieved. However, far less is known about how memories with different functional roles influence response quality. Specifically, how they shape an agent's responses under varying conversational contexts and whether they lead to substantively different response behaviors. Existing evaluations in conversational system are also largely reference-based, insufficiently capturing the nuances in responses that may address users' preferences differently. In this work, we probe the impact of different memory types in shaping agents' responses. We present a fine-grained taxonomy of conversational memory, classify retrieved memories into different role types, and design a user-centric evaluation framework that simulates user perspectives. Through comparative experiments on long-term datasets and frontier LLMs, our analysis reveal many differentiated effects of memories: e.g., clarifying memory improves responses' factual accuracy and constraint awareness, making them more correct and personalized; irrelevant memory reduces topic relevance and degrades constraint awareness. Despite the power of frontier LLMs, these findings shed light on how different memory types can be leveraged to produce more personalized responses and inspire further research in this direction.

Problem

• memory roles
• conversational agents
• response quality
• user preferences
• evaluation framework

Innovation

• memory roles
• conversational agents
• user-centric evaluation
• RAG
• response personalization