Power Grid Control with Graph-Based Distributed Reinforcement Learning

2025-09-02

AI Summary

To address real-time control challenges arising from high renewable energy penetration and expanding grid scale, this paper proposes a distributed reinforcement learning framework based on Graph Neural Networks (GNNs). The method innovatively decomposes both observation and action spaces in tandem, leveraging GNNs to construct localized, structured environmental representations. It initializes the policy via imitation learning and introduces a potential-based reward shaping mechanism to enhance convergence and stability. A hierarchical agent architecture enables coordinated line-level low-level control and system-level high-level coordination. Experiments on the Grid2Op platform demonstrate that the proposed approach significantly outperforms mainstream baselines; its computational efficiency exceeds that of simulation-based expert methods by over an order of magnitude. The results validate the framework’s strong scalability, adaptability, and engineering practicality.

Technology Category

• Machine Learning: Reinforcement Learning
• Planning, Routing, and Scheduling: Planning/Scheduling and Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments

Abstract

The necessary integration of renewable energy sources, combined with the expanding scale of power networks, presents significant challenges in controlling modern power grids. Traditional control systems, which are human and optimization-based, struggle to adapt and to scale in such an evolving context, motivating the exploration of more dynamic and distributed control strategies. This work advances a graph-based distributed reinforcement learning framework for real-time, scalable grid management. The proposed architecture consists of a network of distributed low-level agents acting on individual power lines and coordinated by a high-level manager agent. A Graph Neural Network (GNN) is employed to encode the network's topological information within the single low-level agent's observation. To accelerate convergence and enhance learning stability, the framework integrates imitation learning and potential-based reward shaping. In contrast to conventional decentralized approaches that decompose only the action space while relying on global observations, this method also decomposes the observation space. Each low-level agent acts based on a structured and informative local view of the environment constructed through the GNN. Experiments on the Grid2Op simulation environment show the effectiveness of the approach, which consistently outperforms the standard baseline commonly adopted in the field. Additionally, the proposed model proves to be much more computationally efficient than the simulation-based Expert method.

Problem

• Integrating renewables and scaling power grids challenges control
• Traditional human and optimization-based control lacks adaptability
• Needs dynamic distributed strategies for real-time management

Innovation

• Graph-based distributed reinforcement learning framework
• GNN encodes topological information for local observations
• Integrates imitation learning and reward shaping