Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks

2026-04-13

AI Summary

This work addresses the challenge that large language models struggle to uniformly extract effective memories across heterogeneous tasks due to significant differences in the types of memory required. To tackle this, the paper formally defines the heterogeneous memory extraction task and introduces CluE, a cluster-based self-evolution approach that groups samples by memory context via clustering, optimizes memory extraction prompts independently for each cluster, and dynamically refines them by integrating cross-cluster insights. The authors also construct BEHEMOTH, the first benchmark for evaluating heterogeneous memory extraction. Experimental results demonstrate that CluE achieves unified memory extraction across 18 diverse datasets and outperforms existing self-evolution methods by 9.04% on BEHEMOTH, substantially improving generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Evolutionary Computation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As LLM-based assistants become persistent and personalized, they must extract and retain useful information from past conversations as memory. However, the types of information worth remembering vary considerably across tasks. We formalize the \textit{heterogeneous memory extraction} task and introduce \textbf{BEHEMOTH}, a benchmark that repurposes 18 existing datasets spanning personalization, problem-solving, and agentic tasks, using a downstream utility-driven metric for systematic evaluation. Our empirical analysis confirms that no single static extraction prompt dominates across all task categories, and that existing self-evolving prompt optimization frameworks, originally designed for homogeneous distributions, degrade when training tasks are heterogeneous. To address this, we propose \textbf{CluE}, a cluster-based self-evolving strategy that groups training examples into clusters by extraction scenarios, analyzes each cluster independently, and synthesizes cross-cluster insights to update the extraction prompt. Experiments on BEHEMOTH show that CluE generalizes effectively across heterogeneous tasks ($+$9.04\% relative gain), consistently outperforming prior self-evolving frameworks.

Problem

• heterogeneous memory extraction
• self-evolving LLM
• memory extraction
• prompt optimization
• task heterogeneity

Innovation

• heterogeneous memory extraction
• self-evolving prompts
• cluster-based optimization
• BEHEMOTH benchmark
• LLM memory