Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

2026-05-19

AI Summary

This work addresses critical limitations in existing memory systems for large language model (LLM) agents, which often rely on atomic fact extraction and thereby lose nuanced dialogue details, constrain reasoning capabilities, and struggle to adapt to diverse conversational styles through static prompts. To overcome these issues, the authors propose TriMem, a novel memory framework that introduces a tri-granularity collaborative mechanism—simultaneously preserving raw dialogue excerpts, atomic facts, and synthesized user profiles—to enable high-fidelity storage, efficient retrieval, and deep reasoning. TriMem further incorporates TextGrad-driven prompt optimization and source-aware dialogue anchoring, facilitating parameter-free, lifelong memory evolution. Evaluated on the LoCoMo and PerLTQA benchmarks, TriMem consistently outperforms strong baselines across multiple mainstream LLMs.

Abstract

To enable reliable long-term interaction, LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep reasoning over scattered isolated facts. Moreover, static prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning. We further adopt TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The code is available at https://TMLR-TriMem.github.io .

Problem

• LLM agent memory
• atomic facts
• dialogue history
• memory granularity
• lifelong reasoning

Innovation

• TriMem
• lifelong memory
• multi-granularity representation
• prompt optimization
• LLM agent