Latent Personal Memory: Represent personal memory as dynamic soft prompts

2026-06-18

AI Summary

This work addresses the challenge of efficiently encoding users’ long-term behavioral patterns while keeping large language models frozen. It proposes the Latent Personal Memory (LPM) framework, which compresses user history into N interpretable latent memory slots and employs a shared cross-attention network to generate input-conditioned dynamic soft prompts that are prepended to the frozen model. LPM is the first approach to model user memory as interpretable, dynamic soft prompts, significantly enhancing personalization performance and inference efficiency without fine-tuning the base model. Experiments show that LPM outperforms LoRA and Prompt Tuning by 8.8% and 54.4% in accuracy on PersonaMem v1, respectively, while reducing KV cache usage by 64×. On LoCoMo, LPM achieves parity with LoRA using only 1/120 of its parameters and surpasses full-context methods even within 128K-token contexts.

Abstract

Personalizing large language models (LLMs) requires encoding long-term, user-specific behavioral patterns in a way that is computationally efficient, scalable, and compatible with a frozen base model. We present Latent Personal Memory (LPM), a scalable framework that represents user-specific history as a compact, persistent matrix of N latent slots, that are interpretable. A shared cross-attention projection network maps these slots into dynamic, input-conditioned soft prompts that are prepended to the input of a frozen LLM. We evaluate LPM on PersonaMem v1 and LoCOMO benchmarks across Qwen3-1.7B, 4B, and 8B backbones. Results demonstrate that LPM outperforms LoRA and Prompt Tuning by up to 8.8% and 54.4% in overall accuracy respectively on PersonaMem v1, while reducing KV-cache usage by over 64x. On LoCoMo, LPM matches LoRA accuracy with 120x fewer trainable parameters. We also show that the efficiency of LPM grows with context length and outperforms full-context at 128K context length.

Problem

• personalization
• large language models
• computational efficiency
• scalability
• frozen base model

Innovation

• Latent Personal Memory
• soft prompts
• personalization
• parameter-efficient tuning
• cross-attention projection