Experiential Reinforcement Learning

2026-02-15

AI Summary

This work addresses the challenge in reinforcement learning where sparse and delayed environmental feedback hinders effective behavioral adaptation in language models. To overcome this, the authors propose an experience-reflection-consolidation training paradigm that integrates an explicit self-reflection mechanism directly into policy training. After an initial attempt, the agent receives feedback, generates a structured reflection, and uses this introspection to guide subsequent actions—efficiently translating feedback into lasting behavioral improvements without incurring additional inference overhead. Experimental results demonstrate that the proposed framework significantly outperforms existing reinforcement learning baselines, achieving performance gains of up to 81% in multi-step complex environments and 11% in tool-use reasoning tasks.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.

Problem

• reinforcement learning
• sparse reward
• language models
• delayed feedback
• behavioral improvement

Innovation

• Experiential Reinforcement Learning
• self-reflection
• sparse-reward environments
• behavioral consolidation
• policy training