From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

2026-06-16

AI Summary

This work proposes the LLM-as-Environment-Engineer framework, which leverages a large language model (Qwen3-4B) as an “environment engineer” to dynamically optimize reinforcement learning training environments. Addressing the common reliance on manually tuned settings and the absence of automated, performance-driven environment adaptation, the framework analyzes policy failure trajectories, behavioral summaries, and environmental statistics to automatically reconstruct a multi-dimensional, configurable MAPF-FrozenLake environment. Experimental results demonstrate that reinforcement learning agents fine-tuned within this closed-loop, LLM-driven optimization process not only develop enhanced self-diagnostic capabilities to guide environmental refinements but also achieve superior overall performance—outperforming both larger closed-source models such as GPT and Gemini and fixed-environment baselines—thereby validating the efficacy and novelty of dynamic environment design in reinforcement learning.

Abstract

Reinforcement learning pipelines for Large Language Model (LLM) training often rely on manually redesigned environments between stages, requiring practitioners to heuristically infer which configuration will best improve the current policy. To automate this process, we propose the LLM-as-Environment-Engineer framework in which the current policy model analyzes failure trajectories together with contextual information and proposes modifications to the next-stage training environment configuration. We also introduce MAPF-FrozenLake, a controllable testbed whose generator exposes multi-dimensional environment configurations, making it suitable for studying and benchmarking environment redesign. On this testbed, we condition the environment engineer on structured summaries of policy behavior, failure cases, and environment statistics, from which it produces the configuration for the next training stage. With Qwen3-4B as the backbone, our framework achieves the strongest aggregate performance on our benchmarks, outperforming larger proprietary LLMs (e.g., GPT, Gemini) and fixed-environment training baselines. We further analyze which forms of context are most effective, finding that successful environment updates rely on failure evidence and preserve configurations that already work. Interestingly, the current RL checkpoint serves as a better environment engineer than the original base model, suggesting that policy learning improves the model's ability to diagnose its remaining weaknesses.

Problem

• Reinforcement Learning
• Large Language Models
• Environment Design
• Multi-Agent Reasoning
• Training Automation

Innovation

• LLM-as-Environment-Engineer
• automatic environment redesign
• multi-agent reinforcement learning
• MAPF-FrozenLake
• failure-aware training