Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization

2026-03-25

AI Summary

This work addresses the challenge that existing reinforcement learning approaches for language models struggle to effectively integrate the dual mechanisms of leveraging external experience and internalizing knowledge, both central to human learning. To bridge this gap, we propose the Dual-Guidance Optimization (DGO) framework, which, for the first time, incorporates these mechanisms into reward-verifiable reinforcement learning (RLVR). DGO establishes a closed-loop optimization process by jointly guiding policy exploration through an external experience bank and the model’s internal knowledge. The framework synergistically combines experience retrieval, policy gradient optimization, and trajectory reuse. Extensive experiments across multiple reasoning tasks demonstrate that DGO significantly outperforms current baselines, highlighting the critical role of efficient experience internalization in enhancing the reasoning capabilities of language models.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from verifiable rewards~(RLVR) has emerged as a promising paradigm for reasoning tasks. However, existing RL-based training still remains only a rough approximation to human learning. Human learners leverage both external and internal experience to guide exploration and gradually internalize useful trajectories into stable knowledge. Motivated by this gap, we ask: how can LLMs better utilize and internalize experience during RLVR training? To answer this question, we propose \textbf{D}ual \textbf{G}uidance \textbf{O}ptimization~(\textbf{DGO}), a unified framework that leverages \emph{external} and \emph{internal experience} to improve training effectiveness. Specifically, DGO first constructs an experience bank from previously explored trajectories. The policy then performs exploration under the joint guidance of the experience bank and the model's internal knowledge. The resulting trajectories are further used to refine the experience bank and optimize model parameters, forming a closed loop of experience utilization and internalization. Experiments show that DGO consistently outperforms baseline methods, suggesting that better utilization and internalization of experience lead to more effective reasoning.

Problem

• reinforcement learning
• large language models
• experiential learning
• experience internalization
• reasoning tasks

Innovation

• Dual Guidance Optimization
• Experience Bank
• Reinforcement Learning from Verifiable Rewards
• Experience Internalization
• Large Language Models