SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection

2025-09-24

AI Summary

LLM agents exhibit limited self-reflection capabilities in complex tasks, primarily due to coarse-grained error analysis, reliance on sparse successful trajectories, and poor cross-task experience transfer. To address these limitations, we propose the Multi-level Reflection Synthesis (MRS) framework, which establishes three hierarchical reflection mechanisms: single-trajectory correction, intra-task attribution, and inter-task pattern transfer—augmented by prospective reflection to enable dynamic, interactive adjustment. Methodologically, MRS leverages a retrospective language model integrated with fine-grained error localization, structured error categorization, and cross-task failure pattern mining. Evaluated on three benchmarks—TravelPlanner, NATURAL PLAN, and Tau-bench—MRS significantly outperforms existing reflection baselines, demonstrating superior error diagnosis accuracy, sustained performance improvement across iterations, and robust generalization to unseen tasks.

Technology Category

• Multiagent Systems: Multiagent Planning
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Despite the rapid advancements in LLM agents, they still face the challenge of generating meaningful reflections due to inadequate error analysis and a reliance on rare successful trajectories, especially in complex tasks. In this work, we propose SAMULE, a new framework for self-learning agents powered by a retrospective language model that is trained based on Multi-Level Reflection Synthesis. It first synthesizes high-quality reflections across three complementary levels: Single-Trajectory Learning (micro-level) for detailed error correction; Intra-Task Learning (meso-level) to build error taxonomies across multiple trials of the same task, and Inter-Task Learning (macro-level) to extract transferable insights based on same typed errors from diverse task failures. Then we fine-tune a language model serving as the retrospective model to generate reflections during inference. We further extend our framework to interactive settings through a foresight-based reflection mechanism, enabling agents to proactively reflect and adapt during user interactions by comparing predicted and actual responses. Extensive experiments on three challenging benchmarks - TravelPlanner, NATURAL PLAN, and Tau-bench - demonstrate that our approach significantly outperforms reflection-based baselines. Our results highlight the critical role of well-designed reflection synthesis and failure-centric learning in building self-improving LLM agents.

Problem

• LLM agents struggle with meaningful reflection due to inadequate error analysis
• Agents rely on rare successful trajectories especially in complex tasks
• Current approaches lack multi-level reflection synthesis for self-improvement

Innovation

• Multi-level reflection synthesis across micro meso macro
• Fine-tuned retrospective model for generating reflections
• Foresight-based reflection mechanism for interactive adaptation