Syllabus: Portable Curricula for Reinforcement Learning Agents

2024-11-18

arXiv.org

AI Summary

Current reinforcement learning (RL) frameworks lack native, non-intrusive support for curriculum learning (CL), requiring invasive code modifications to implement. To address this, we propose CLib—the first lightweight, general-purpose curriculum learning library. CLib features a unified API and modular architecture comprising: (i) an environment-agnostic curriculum scheduler, (ii) a distributed sampling adapter, and (iii) a cross-framework bridging layer supporting both PyTorch and TensorFlow backends. It integrates seamlessly with five+ mainstream RL libraries—including Ray RLlib and CleanRL—without altering underlying training logic. We demonstrate the first successful application of CL in complex environments NetHack and Neural MMO, and validate CLib across nine benchmark tasks, consistently outperforming state-of-the-art baselines. By eliminating implementation barriers, CLib lowers the entry threshold for CL adoption, promotes standardization, and enhances reproducibility in RL research.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Consent frameworks and practices on the web

Abstract

Curriculum learning has been a quiet yet crucial component of many of the high-profile successes of reinforcement learning. Despite this, none of the major reinforcement learning libraries directly support curriculum learning or include curriculum learning implementations. These methods can improve the capabilities and robustness of RL agents, but often require significant, complex changes to agent training code. We introduce Syllabus, a library for training RL agents with curriculum learning, as a solution to this problem. Syllabus provides a universal API for curriculum learning algorithms, implementations of popular curriculum learning methods, and infrastructure for easily integrating them with distributed training code written in nearly any RL library. Syllabus provides a minimal API for each of the core components of curriculum learning, dramatically simplifying the process of designing new algorithms and applying existing algorithms to new environments. We demonstrate that the same Syllabus code can be used to train agents written in multiple different RL libraries on numerous domains. In doing so, we present the first examples of curriculum learning in NetHack and Neural MMO, two of the premier challenges for single-agent and multi-agent RL respectively, achieving strong results compared to state of the art baselines.

Problem

• Lack of direct support for curriculum learning in major RL libraries
• Complex code changes needed for curriculum learning methods
• Difficulty in adapting curriculum learning to new environments

Innovation

• Universal API for curriculum learning
• Modular automatic curriculum methods
• Integration with asynchronous RL training