EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning

2026-06-24

AI Summary

This work addresses the challenges in reinforcement learning where Actor-Critic architectures rely heavily on manual design, automated search suffers from low efficiency, and the design space remains largely unstructured. To overcome these limitations, the authors propose EVOM, a novel framework that introduces, for the first time, a large language model (LLM)-based design agent fully decoupled from policy execution. The architecture search is formulated as a bilevel optimization problem: the inner loop employs low-fidelity proximal policy optimization (PPO) for efficient weight training, while the outer loop leverages an LLM-driven programmatic representation combined with an evolutionary mechanism to iteratively refine architectures. Evaluated on Ant-v4 and HalfCheetah-v4, EVOM outperforms handcrafted baselines, LLM-based random search, and the state-of-the-art MLES method. Ablation studies further confirm the critical contributions of both the meta-evolutionary loop and the LLM design agent.

Abstract

In actor-critic reinforcement learning, network architectures are typically manually designed. Automating this design is challenging because each candidate must be trained before evaluation, and the design space is open-ended. To address these challenges, we introduce EVOM, an agentic meta-evolution framework for discovering high-performance actor-critic architectures. We frame architecture search as a bi-level optimization: an inner loop trains weights via the low-fidelity proximal policy optimization (PPO), while an outer loop drives meta-evolution by iteratively refining architecture programs. Crucially, this outer loop is powered by an LLM-based design agent that operates purely as an architecture designer, completely decoupled from policy execution and environment control. Experiments reveal that EVOM outperforms the manually designed baseline, an LLM-guided random search, and the state-of-the-art LLM-guided programmatic policy search method MLES, delivering superior performance on Ant-v4 and HalfCheetah-v4. Ablation studies validate that both the meta-evolution loop and the LLM Design Agent are indispensable for final performance.

Problem

• actor-critic
• architecture search
• reinforcement learning
• automated design
• open-ended design space

Innovation

• agentic meta-evolution
• actor-critic architecture search
• LLM-based design agent
• bi-level optimization
• automated reinforcement learning