AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models

2026-04-19

AI Summary

This work addresses the challenge that large language models often lose critical contextual and retrieval cues during extended interactions due to ineffective memory mechanisms and frequent overwriting of interaction history. Inspired by the Proust phenomenon, the authors propose AnchorMem, a novel memory framework that decouples retrieval units from generative context. AnchorMem extracts atomic facts from interaction history as immutable anchors, preserving their original contextual embeddings while constructing a higher-order event graph to integrate multi-fact representations. By eschewing generic entity-based or summary-based reconstructions, the approach achieves both fine-grained retrievability and high contextual fidelity. Experimental results on the LoCoMo benchmark demonstrate that AnchorMem significantly outperforms existing memory-augmented methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While large language models have achieved remarkable performance in complex tasks, they still need a memory system to utilize historical experience in long-term interactions. Existing memory methods (e.g., A-Mem, Mem0) place excessive emphasis on organizing interactions by frequently rewriting them, however, this heavy reliance on summarization risks diluting essential contextual nuances and obscuring key retrieval features. To bridge this gap, we introduce AnchorMem, a novel memory framework inspired by the Proust Phenomenon in cognitive science, where a specific anchor triggers a holistic recollection. We propose a method that decouples the retrieval unit from the generation context. AnchorMem extracts atomic facts from interaction history to serve as retrieval anchors, while preserving the original context as the immutable context. To reveal implicit narrative cues, we construct an associative event graph that uses higher-order event links that bind sets of related facts into shared event representations, strengthening cross-memory integration without relying on generic entities as bridges. During retrieval, the system anchors queries to specific facts and events to locate relevant memories, but reconstructs the context using the associated raw chunks and events. Our method reconciles fine-grained retrieval with the contextual integrity of interactions. Experiments across three closed-source and open-source models on the LoCoMo benchmark demonstrate that AnchorMem significantly outperforms baselines. Code is available at https://github.com/RayNeo-AI-2025/AnchorMem.

Problem

• memory system
• contextual integrity
• retrieval features
• long-term interactions
• large language models

Innovation

• AnchorMem
• memory framework
• associative event graph
• atomic facts
• context integrity