CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zero

2025-05-24

AI Summary

This study addresses three key challenges in circular economy (CE) research: (1) difficulty in optimizing solid/fluid material circulation, (2) weak interdisciplinary modeling capabilities, and (3) high barriers to intelligent decision-making. To this end, we propose the first thermodynamics-driven deep reinforcement learning (DRL) framework explicitly designed for net-zero objectives. Methodologically, we pioneer the integration of thermodynamic material networks with state-space dynamic modeling into DRL environment design—yielding a physically interpretable, scalable, and intelligent optimization environment. Implementation leverages Stable-Baselines3, supporting mainstream algorithms including PPO and SAC, and is fully open-source and Colab-executable. Our core contribution is CiRL: the first open-source DRL toolkit tailored specifically for CE researchers. Validated across diverse material circulation scenarios, CiRL significantly lowers the barrier to intelligent modeling and decision-making for sustainable systems, thereby advancing reproducible, interdisciplinary CE research.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The demand of finite raw materials will keep increasing as they fuel modern society. Simultaneously, solutions for stopping carbon emissions in the short term are not available, thus making the net zero target extremely challenging to achieve at scale. The circular economy (CE) paradigm is gaining attention as a solution to address climate change and the uncertainties of supplies of critical materials. Hence, in this paper, we introduce CiRL, a deep reinforcement learning (DRL) library of environments focused on the circularity of both solid and fluid materials. The integration of DRL into the design of material circularity is possible thanks to the formalism of thermodynamical material networks, which is underpinned by compartmental dynamical thermodynamics. Along with the focus on circularity, this library has three more features: the new CE-oriented environments are in the state-space form, which is typically used in dynamical systems analysis and control designs; it is based on a state-of-the-art Python library of DRL algorithms, namely, Stable-Baselines3; and it is developed in Google Colaboratory to be accessible to researchers from different disciplines and backgrounds as is often the case for circular economy researchers and engineers. CiRL is publicly available.

Problem

• Addressing finite raw material demand and net zero challenges
• Promoting circular economy via reinforcement learning environments
• Integrating thermodynamics with DRL for material circularity solutions

Innovation

• DRL library for material circularity
• Thermodynamical material networks formalism
• Accessible via Google Colaboratory platform