Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models

2026-03-30

AI Summary

This work proposes a large language model (LLM)-driven evolutionary framework that directly searches for complete reinforcement learning update rules, moving beyond manually designed, fixed mechanisms. By excluding canonical structures such as actor-critic architectures, temporal difference learning, and value bootstrapping, the approach employs an LLM as a mutation operator within the REvolve system to end-to-end evolve novel algorithms. A post-evolution phase further automates the optimization of hyperparameter ranges. This study represents the first extension of LLM-based evolutionary strategies from reward function discovery to the full discovery of reinforcement learning algorithms. Evaluated across multiple Gymnasium benchmarks, the evolved algorithms match or rival established baselines—including SAC, PPO, DQN, and A2C—demonstrating the efficacy and potential of unconventional learning rules.

Abstract

Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor--critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C.

Problem

• reinforcement learning
• algorithm discovery
• update rules
• evolutionary framework
• large language models

Innovation

• evolutionary algorithm discovery
• large language models
• reinforcement learning
• update rule search
• hyperparameter refinement