Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning

2026-06-15

AI Summary

This work addresses the poor generalization of reinforcement learning (RL) when training and testing environments exhibit distributional shifts—a limitation exacerbated in privacy-sensitive or data-scarce settings where diverse training environments and full trajectory access are unavailable. To overcome this, the authors propose GERS, a novel method that, for the first time, integrates evolutionary algorithms with RL under the constraint of accessing only scalar feedback from validation environments without observing their trajectories. GERS employs a bilevel optimization framework: the lower level learns a policy from limited training environments, while the upper level leverages CMA-ES to optimize reward-shaping parameters for enhanced generalization. Experiments demonstrate that GERS significantly outperforms standard RL baselines across multiple continuous control tasks and achieves generalization performance comparable to domain randomization methods—despite not requiring access to trajectory data.

Abstract

Reinforcement learning (RL) often suffers from performance degradation when deployed in environments that differ from those encountered during training. Existing techniques such as domain randomization (DR) mitigate this, but require access to diverse training environments and full trajectory observability, assumptions that fail in privacy-preserving or restricted scenarios where only scalar performance metrics are available. We propose Generalization via Evolutionary Reward Shaping (GERS), a bilevel optimization approach to improve generalization on unseen test environments using only scalar feedback from validation environments. At the lower level, an RL agent guided via a reward function shaped by the upper level learns a policy on a limited set of training environments with accessible trajectory data; at the upper level, CMA-ES optimizes the reward shaping parameters to maximize the cumulative unshaped reward on separate validation environments for which trajectory access is unavailable. Results on continuous control tasks indicate that GERS outperforms the standard RL baseline on unseen test environments. GERS performance is comparable to DR, despite DR treating the combined set of training and validation environments of GERS as a single training set that requires trajectory access, whereas GERS cannot access validation trajectories. These results confirm that GERS effectively enhances generalization under restricted data access constraints.

Problem

• reinforcement learning
• generalization
• reward shaping
• bilevel optimization
• data access constraints

Innovation

• bilevel optimization
• reward shaping
• generalization
• evolutionary algorithm
• reinforcement learning