Complementary Reinforcement Learning

2026-03-18

AI Summary

This work addresses the sample inefficiency in reinforcement learning caused by sparse rewards and the misalignment between historical experiences and evolving policies. Inspired by the complementary learning systems in neuroscience, the authors propose an end-to-end differentiable architecture that dynamically co-optimizes an experience extractor and a policy executor during training. The experience extractor prioritizes memories based on their contribution to policy improvement, while the policy executor is updated using outcome-based rewards; both components evolve in tandem to prevent capability mismatch. This joint adaptive optimization of experience management and policy learning yields a 10% performance gain in single-task settings and demonstrates strong scalability in multi-task scenarios, marking the first successful integration of adaptive experience replay with policy learning in a unified framework.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.

Problem

• Reinforcement Learning
• sample efficiency
• experience reuse
• actor-experience misalignment
• complementary learning

Innovation

• Complementary Reinforcement Learning
• experience extractor
• co-evolution
• sample efficiency
• LLM-based agents