A New Error Temporal Difference Algorithm for Deep Reinforcement Learning in Microgrid Optimization

2025-11-22

AI Summary

Existing deep reinforcement learning (DRL)-based energy optimization methods for microgrids often neglect the uncertainty inherent in forecasting models, leading to suboptimal control policies. To address this, we propose the Error Temporal Differencing (ETD) algorithm—the first approach to explicitly incorporate the temporal dynamics of prediction errors into a Deep Q-Network (DQN) framework, augmented with a weighted averaging mechanism for uncertainty-aware decision-making. Within a Markov Decision Process (MDP) formulation, ETD explicitly quantifies and mitigates the impact of forecast bias on policy updates, thereby significantly enhancing the robustness and adaptability of DRL in highly volatile microgrid environments. Extensive simulations on a real-world U.S. dataset demonstrate that ETD reduces energy costs by 12.3% and power fluctuation risk by 27.6% compared to baseline methods, while effectively improving coordinated regulation between renewable generation and energy storage systems.

Technology Category

• Planning, Routing, and Scheduling: Temporal Planning
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Predictive control approaches based on deep reinforcement learning (DRL) have gained significant attention in microgrid energy optimization. However, existing research often overlooks the issue of uncertainty stemming from imperfect prediction models, which can lead to suboptimal control strategies. This paper presents a new error temporal difference (ETD) algorithm for DRL to address the uncertainty in predictions,aiming to improve the performance of microgrid operations. First,a microgrid system integrated with renewable energy sources (RES) and energy storage systems (ESS), along with its Markov decision process (MDP), is modelled. Second, a predictive control approach based on a deep Q network (DQN) is presented, in which a weighted average algorithm and a new ETD algorithm are designed to quantify and address the prediction uncertainty, respectively. Finally, simulations on a realworld US dataset suggest that the developed ETD effectively improves the performance of DRL in optimizing microgrid operations.

Problem

• Addresses prediction uncertainty in microgrid DRL control
• Improves optimization of renewable energy and storage systems
• Develops error temporal difference algorithm for better performance

Innovation

• New error temporal difference algorithm for DRL
• Weighted average algorithm quantifies prediction uncertainty
• ETD improves microgrid optimization with renewable energy