M$^\star$: Every Task Deserves Its Own Memory Harness

2026-04-09

AI Summary

Current large language model agents rely on fixed memory architectures that hinder cross-task transfer and limit performance. This work proposes a task-adaptive approach to automatically evolve memory mechanisms by modeling the memory system as an executable Python program comprising a data schema, storage logic, and workflow instructions. The method employs population-based, reflective code evolution to jointly optimize these components. For the first time, it enables automatic customization of memory programs, significantly outperforming fixed-memory baselines across four benchmarks—including dialogue, embodied planning, and expert reasoning—while generating diverse, task-specific memory structures that surpass the limitations of universal memory paradigms.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Evolutionary Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language model agents rely on specialized memory systems to accumulate and reuse knowledge during extended interactions. Recent architectures typically adopt a fixed memory design tailored to specific domains, such as semantic retrieval for conversations or skills reused for coding. However, a memory system optimized for one purpose frequently fails to transfer to others. To address this limitation, we introduce M$^\star$, a method that automatically discovers task-optimized memory harnesses through executable program evolution. Specifically, M$^\star$ models an agent memory system as a memory program written in Python. This program encapsulates the data Schema, the storage Logic, and the agent workflow Instructions. We optimize these components jointly using a reflective code evolution method; this approach employs a population-based search strategy and analyzes evaluation failures to iteratively refine the candidate programs. We evaluate M$^\star$ on four distinct benchmarks spanning conversation, embodied planning, and expert reasoning. Our results demonstrate that M$^\star$ improves performance over existing fixed-memory baselines robustly across all evaluated tasks. Furthermore, the evolved memory programs exhibit structurally distinct processing mechanisms for each domain. This finding indicates that specializing the memory mechanism for a given task explores a broad design space and provides a superior solution compared to general-purpose memory paradigms.

Problem

• memory system
• task-specific optimization
• large language model agents
• knowledge reuse
• transfer limitation

Innovation

• memory program evolution
• task-specific memory
• reflective code evolution
• LLM agent memory
• executable memory harness