Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval

2026-03-10

AI Summary

This work addresses the challenge of balancing efficiency and depth in memory retrieval for personalized large language models, where existing approaches either suffer from high computational overhead or rely solely on shallow similarity matching. Inspired by human dual-process memory mechanisms, we propose RF-Mem—the first framework to integrate a familiarity-recollection dual-path architecture into LLM memory retrieval. The system adaptively selects between a fast familiarity path and a recollection path based on familiarity signals (mean and entropy) in the embedding space; the latter iteratively expands evidence through α-mixing of memory clusters and the query to reconstruct contextual information. Experiments across three benchmarks and varying corpus scales demonstrate that RF-Mem significantly outperforms single-retrieval and full-context inference methods under fixed computational budgets and latency constraints.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Personalized large language models (LLMs) rely on memory retrieval to incorporate user-specific histories, preferences, and contexts. Existing approaches either overload the LLM by feeding all the user's past memory into the prompt, which is costly and unscalable, or simplify retrieval into a one-shot similarity search, which captures only surface matches. Cognitive science, however, shows that human memory operates through a dual process: Familiarity, offering fast but coarse recognition, and Recollection, enabling deliberate, chain-like reconstruction for deeply recovering episodic content. Current systems lack both the ability to perform recollection retrieval and mechanisms to adaptively switch between the dual retrieval paths, leading to either insufficient recall or the inclusion of noise. To address this, we propose RF-Mem (Recollection-Familiarity Memory Retrieval), a familiarity uncertainty-guided dual-path memory retriever. RF-Mem measures the familiarity signal through the mean score and entropy. High familiarity leads to the direct top-K Familiarity retrieval path, while low familiarity activates the Recollection path. In the Recollection path, the system clusters candidate memories and applies alpha-mix with the query to iteratively expand evidence in embedding space, simulating deliberate contextual reconstruction. This design embeds human-like dual-process recognition into the retriever, avoiding full-context overhead and enabling scalable, adaptive personalization. Experiments across three benchmarks and corpus scales demonstrate that RF-Mem consistently outperforms both one-shot retrieval and full-context reasoning under fixed budget and latency constraints. Our code can be found in the Reproducibility Statement.

Problem

• personalized LLMs
• memory retrieval
• recollection
• familiarity
• adaptive retrieval

Innovation

• Recollection-Familiarity
• Adaptive Retrieval
• Memory Personalization
• Uncertainty-Guided
• Dual-Process Memory