Dynamics Models for Offline Hyperparameter Selection in Real-World RL

2026-08-11

AI Summary

This work addresses the challenge of high online tuning costs and the absence of simulators in real-world reinforcement learning systems by proposing a method that constructs a calibrated model from offline data to approximate environment dynamics, enabling offline hyperparameter selection. The approach is applied for the first time to a municipal water treatment plant, employing a k-nearest neighbors model with Laplacian distance for nexting prediction. Evaluated on high-dimensional, non-stationary data over annual timescales, the method demonstrates strong scalability and robustness to distributional shifts. Experimental results show that the calibrated model generates realistic long-horizon trajectories, accurately reproduces hyperparameter sensitivity trends, and effectively supports fine-grained tuning of the agent’s learning rate.

Abstract

A key obstacle to deploying reinforcement learning in real-world systems is hyperparameter selection, particularly when simulators are unavailable and online experimentation is costly. Prior work has proposed calibration models trained on offline data to approximate environment dynamics and enable offline hyperparameter selection, but these methods have so far been evaluated only in simple simulated settings. In this paper, we present the first application of calibration models in a real-world industrial setting: a municipal water treatment plant. We evaluate several calibration model approaches, including a k-nearest neighbors model with a Laplacian distance metric, on high-dimensional, non-stationary sensor data for nexting prediction tasks. Our results show that these models can generate realistic long-horizon rollouts and recover meaningful hyperparameter sensitivity trends. We further examine how calibration models scale to year-long datasets, how they support the selection of fine-tuning learning rates for pre-trained agents, and how robust they are under distribution shift. Overall, our findings provide a proof of concept for using offline dynamics models to support RL deployment in real-world environments, while highlighting important practical challenges for future work.

Problem

• offline hyperparameter selection
• real-world reinforcement learning
• dynamics models
• calibration models
• distribution shift

Innovation

• calibration models
• offline hyperparameter selection
• real-world reinforcement learning
• non-stationary sensor data
• long-horizon rollouts