State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

2026-06-25

AI Summary

This study addresses the critical role of state representation in reinforcement learning–based arbitrage for pumped hydro storage. Under a fixed Double DQN architecture, the authors systematically evaluate the impact of three classes of state features—absolute prices, relative historical prices, and short-term price forecasts—and their combinations on trading performance. Experiments conducted in the HydroDam environment using electricity price data from Belgium and multiple ENTSO-E regions demonstrate that state representations integrating multi-source features significantly outperform those based on single features or heuristic approaches, achieving an average test-set return of 55.6% and a median of 47.5% across 39 regions. The findings underscore state representation as a cornerstone of effective energy storage trading strategies and confirm that fusing diverse predictive signals enhances cross-market generalization.

Abstract

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.

Problem

• state representation
• deep reinforcement learning
• energy trading
• feature design
• policy transfer

Innovation

• state representation
• deep reinforcement learning
• energy trading
• feature engineering
• transferability