Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

2026-08-14

AI Summary

This study addresses the challenge of fixed environment generation strategies failing to adaptively align with learning frontiers in reinforcement learning for terminal agents. We propose Envs-FORGE, a novel framework introducing a seed-wise dynamic environment synthesis mechanism guided by learning frontiers. This approach translates validator rewards into synthetic actions and optimizes generation policies via mixed-integer linear programming while synchronously reconstructing test environments, retaining only gold-standard data for training. Experiments on Qwen3.5-35B demonstrate that Envs-FORGE improves tb-core Pass@1 by 9.2% and achieves 77.1% on SWE-bench Verified, significantly outperforming fixed-strategy baselines. These results confirm the framework’s effectiveness in overcoming the limitations of conventional prompting strategies by enabling adaptive curriculum generation tailored to evolving model capabilities.

Abstract

Reinforcement learning (RL) for terminal agents needs executable training environments with reliable rewards and useful difficulty. Fixed recipes such as few-shot, Self-Instruct, and Evol-Instruct apply the same prompting policy to every seed, even when the current policy would benefit from a harder, easier, or simply different task. We present Envs-FORGE, a prompting policy that converts verifier rewards into per-seed environment-synthesis actions. Envs-FORGE estimates seed pass rates, scores six projection--direction actions around a target learning frontier, and solves a per-seed mixed-integer linear program (MILP) to choose the action that conditions generation. The selected action drives synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment; only gold-verified bundles enter RL training. The indexed MILP form also supports optional soft skill coverage for portfolio planning. On Qwen 3.5 35B, Envs-FORGE improves Pass@1 over Base by 9.2 percentage points on tb-core (40.0% to 49.2%) and 6.4 points on tb-2.0 (23.0% to 29.4%), exceeding the strongest fixed-recipe baseline by 2.4 and 2.1 points. It reaches 77.1% on SWE-bench Verified versus 73.4% for Base, and improves tb-core by 6.8--9.2 points across the evaluated 4B--35B models. All synthesis methods export 100 verified environments and use 2.27M--2.88M synthesis tokens, placing the comparison at the same downstream training-set size and the same operational scale. The source code is available at https://github.com/DataArcTech/DataArc-SynData-Toolkit/.

Problem

• Reinforcement Learning
• Environment Synthesis
• Terminal Agents
• Reward Grounding
• Adaptive Difficulty

Innovation

• Environment Synthesis
• Mixed-Integer Linear Programming
• Reward-Grounded Adaptation
• Learning Frontier
• Agent Reinforcement Learning