Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination

2026-07-14

AI Summary

This work addresses the safety limitations of standard federated reinforcement learning approaches, such as FedAvg, in microgrid energy coordination, where system constraints are typically ignored, leading to potentially unsafe global policies. To overcome this issue, the authors propose a lightweight penalty-based aggregation mechanism that operates at the server level, combining local rewards and constraint violation estimates using weights proportional to \( R_i - \alpha V_i \). This approach ensures safety without requiring dual optimization or modifications to local training procedures. Empirical evaluations on the DairyGridEnv simulation environment, using real-world load data from Finland and Germany, demonstrate that the proposed method consistently reduces constraint violations and improves cumulative rewards across both synthetic and real-world scenarios, outperforming conventional FedAvg.

Abstract

Federated Reinforcement Learning (FedRL) enables coordination of distributed energy resources without sharing raw local data, but standard aggregation methods such as FedAvg do not account for system-level constraints, often leading to unsafe global behavior. In this work, we study constraint-aware aggregation for federated reinforcement learning in distributed energy coordination. We propose aggregation rules that incorporate both local performance and estimated constraint violation into the server-side update. Among these, a simple penalty-based rule, $w_i \propto R_i - αV_i$, consistently provides the most reliable trade-off between reward and safety, without requiring dual optimization or modifications to local training. \textcolor{black}{We evaluate our approach on DairyGridEnv, a benchmark modeling multiple farms coordinating battery storage under stochastic demand and a shared grid capacity constraint, and further assess robustness using real load-driven demand profiles from Finland and the German FIELD dataset. Across multiple seeds, penalty-based aggregation substantially reduces violations while improving reward relative to FedAvg in both synthetic and real load-driven settings.} A combined reward-violation scheme exposes a tunable trade-off via $λ$, but is less stable. These results demonstrate that lightweight aggregation strategies can substantially improve empirical safety in federated reinforcement learning while preserving standard communication protocols.

Problem

• Federated Reinforcement Learning
• Constraint Violation
• Energy Coordination
• Microgrid
• Safe Aggregation

Innovation

• Constraint-Aware Aggregation
• Federated Reinforcement Learning
• Microgrid Coordination
• Penalty-Based Aggregation
• Safety in RL