RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning

2024-05-29

arXiv.org

AI Summary

High-cost and sparse extrinsic rewards in complex environments hinder effective unsupervised reinforcement learning (RL). To address this, we propose IRL-Bench—the first unified, modular, plug-and-play intrinsic reward framework. Implemented in PyTorch, it systematically integrates eight state-of-the-art intrinsic motivation methods—including prediction error, information gain, state counting, and dynamics modeling—while explicitly documenting critical implementation details and establishing standardized best practices for the first time. The framework supports parallel multi-algorithm training and modular component replacement, enabling open, reproducible, and fair cross-method evaluation. Empirical validation on Atari and MiniGrid confirms both reproducibility and sensitivity to algorithmic differences. By lowering barriers to intrinsic RL research, IRL-Bench advances community-wide standardization and comparability of intrinsic reward methods.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Extrinsic rewards can effectively guide reinforcement learning (RL) agents in specific tasks. However, extrinsic rewards frequently fall short in complex environments due to the significant human effort needed for their design and annotation. This limitation underscores the necessity for intrinsic rewards, which offer auxiliary and dense signals and can enable agents to learn in an unsupervised manner. Although various intrinsic reward formulations have been proposed, their implementation and optimization details are insufficiently explored and lack standardization, thereby hindering research progress. To address this gap, we introduce RLeXplore, a unified, highly modularized, and plug-and-play framework offering reliable implementations of eight state-of-the-art intrinsic reward methods. Furthermore, we conduct an in-depth study that identifies critical implementation details and establishes well-justified standard practices in intrinsically-motivated RL. Our documentation, examples, and source code are available at https://github.com/RLE-Foundation/RLeXplore.

Problem

• Extrinsic rewards lack efficiency in complex RL environments
• Intrinsic reward methods lack standardization and implementation details
• Need unified framework for intrinsic reward research in RL

Innovation

• Unified framework for intrinsic reward methods
• Modularized plug-and-play implementation design
• Standardized practices in motivated RL