Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinement

2025-12-21

AI Summary

Large language model (LLM) agents face challenges in achieving sample-efficient, interpretable, and continual evolution without parameter updates. Method: This paper proposes a purely external, hierarchical program memory system that extracts reusable program modules from execution trajectories, integrates Bayesian reliability modeling with expected-utility-driven action selection, and introduces a success/failure contrastive refinement mechanism for zero-parameter-update autonomous learning. Contribution/Results: We pioneer the “frozen-LLM + external program memory” paradigm, unifying program abstraction compression and Bayesian decision-making. Evaluated on four benchmarks, our approach achieves an average performance of 78.1%. On ALFWorld, zero-shot generalization to unseen tasks improves to 90.3% (+3.1%), memory construction accelerates by 2800×, and 2,851 trajectories are compressed into 187 high-efficiency programs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present MACLA, a framework that decouples reasoning from learning by maintaining a frozen large language model while performing all adaptation in an external hierarchical procedural memory. MACLA extracts reusable procedures from trajectories, tracks reliability via Bayesian posteriors, selects actions through expected-utility scoring, and refines procedures by contrasting successes and failures. Across four benchmarks (ALFWorld, WebShop, TravelPlanner, InterCodeSQL), MACLA achieves 78.1 percent average performance, outperforming all baselines. On ALFWorld unseen tasks, MACLA reaches 90.3 percent with 3.1 percent positive generalization. The system constructs memory in 56 seconds, 2800 times faster than the state-of-the-art LLM parameter-training baseline, compressing 2851 trajectories into 187 procedures. Experimental results demonstrate that structured external memory with Bayesian selection and contrastive refinement enables sample-efficient, interpretable, and continually improving agents without LLM parameter updates.

Problem

• Enables sample-efficient agents without LLM parameter updates
• Extracts reusable procedures from trajectories via Bayesian selection
• Refines procedures by contrasting successes and failures

Innovation

• External hierarchical procedural memory for LLM agents
• Bayesian posterior tracking for reliability and selection
• Contrastive refinement of procedures from successes and failures