Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systems

2025-11-25

AI Summary

To address the challenge of jointly optimizing economic efficiency and reliability in multi-microgrid systems (MMSs) under high penetration of distributed renewable energy, this paper proposes a distributed independent policy gradient reinforcement learning method. We first introduce mean–variance team stochastic games into MMS energy management—establishing a novel optimization framework that jointly minimizes operational cost and quantifies power fluctuation risk. Second, we design a decentralized independent policy gradient algorithm with theoretical convergence guarantees, integrating deep reinforcement learning and distributional parameter estimation to enable data-driven policy learning in both model-known and model-unknown settings. Experimental results across various system scales demonstrate significant reductions in grid procurement costs and power fluctuations, thereby enhancing overall economic performance and operational stability while fully unlocking the potential of distributed collaborative optimization.

Technology Category

• Multiagent Systems: Multiagent Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Distributed Machine Learning & Federated Learning

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Efficiency and reliability are both crucial for energy management, especially in multi-microgrid systems (MMSs) integrating intermittent and distributed renewable energy sources. This study investigates an economic and reliable energy management problem in MMSs under a distributed scheme, where each microgrid independently updates its energy management policy in a decentralized manner to optimize the long-term system performance collaboratively. We introduce the mean and variance of the exchange power between the MMS and the main grid as indicators for the economic performance and reliability of the system. Accordingly, we formulate the energy management problem as a mean-variance team stochastic game (MV-TSG), where conventional methods based on the maximization of expected cumulative rewards are unsuitable for variance metrics. To solve MV-TSGs, we propose a fully distributed independent policy gradient algorithm, with rigorous convergence analysis, for scenarios with known model parameters. For large-scale scenarios with unknown model parameters, we further develop a deep reinforcement learning algorithm based on independent policy gradients, enabling data-driven policy optimization. Numerical experiments in two scenarios validate the effectiveness of the proposed methods. Our approaches fully leverage the distributed computational capabilities of MMSs and achieve a well-balanced trade-off between economic performance and operational reliability.

Problem

• Optimizing economic and reliable energy management in multi-microgrid systems
• Developing distributed algorithms for mean-variance team stochastic games
• Balancing operational costs and reliability with renewable energy integration

Innovation

• Independent policy gradient algorithm for distributed energy management
• Mean-variance team stochastic game modeling economic reliability
• Deep reinforcement learning for large-scale unknown parameter scenarios