LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents

2026-06-16

AI Summary

This work addresses the strong dataset dependency in post-training for reinforcement learning, where conventional fixed scheduling strategies struggle to dynamically balance exploration and exploitation and fail to adaptively adjust hyperparameters such as regularization. To overcome this limitation, the authors propose a tree-search framework powered by large language model (LLM) agents that automatically diagnoses trajectory pathologies and jointly optimizes multiple hyperparameters across multi-stage training. The study uncovers, for the first time, structural patterns wherein capacity-related parameters exhibit monotonic accumulation while regularization parameters oscillate, leading to transferable adaptive scheduling principles that uniformly explain the common dynamics of policy behavior across diverse tasks. Evaluated on four GRPO benchmarks, the method achieves performance gains of 9%–140% over baselines, substantially outperforming grid search (+6%–15%), random search, and skill-based agents.

Abstract

RL post-training strategies are dataset-dependent and reveal a recurring empirical pattern: capacity parameters accumulate monotonically across stages, while regularization parameters predominantly oscillate in response to shifting training dynamics. This distinction matters because fixed schedules commit all parameters to fixed trajectories and therefore cannot express the non-stationary exploration-exploitation tradeoffs that regularization must track; the principle provides actionable design rules for multi-stage training. We discover this through LLMZero, a system where LLM agents search over training trajectories via tree search, diagnosing pathologies at each checkpoint and proposing coordinated multi-parameter transitions. Across 4 diverse GRPO tasks, LLMZero discovers strategies that improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, consistently outperforming random search and the skill-based agent. The structural principle transfers across tasks, providing an explanation for why discovered strategies take qualitatively different forms yet share similar parameter dynamics.

Problem

• RL post-training
• training strategies
• regularization parameters
• non-stationary dynamics
• multi-stage training

Innovation

• LLM agents
• adaptive training strategies
• RL post-training
• parameter dynamics
• tree search