Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

2026-06-09

AI Summary

This work addresses the scarcity of frontier tasks in reinforcement learning, where existing task distributions are prone to saturation and naively generated tasks are often overly simple, unsolvable, or ambiguously defined. The authors propose PROPEL, a framework that enables efficient training of a task generator on the learnable frontier without repeatedly invoking a solver. By freezing a reference model and training a lightweight activation probe on a one-time annotated corpus to predict task pass rates, PROPEL replaces costly solver rollouts. Coupled with reinforcement learning, this approach optimizes the generator to produce tasks that remain valid while significantly improving difficulty alignment and training efficiency. Experiments show that on code generation tasks, the proportion of frontier tasks for Qwen2.5-3B and 7B increases from 10.1%/5.3% to 20.0%/12.6%, respectively; on SWE tasks, Qwen3.5-27B achieves a rise from 9.8% to 19.6% in target pass-rate tasks on unseen repositories.

Abstract

The limiting resource for training agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes; solver-in-the-loop generator training is intractable. We introduce PROPEL, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from $10.1\% \rightarrow 20.0\%$ for a Qwen2.5-3B-Instruct solver and from $5.3\% \rightarrow 12.6\%$ for a Qwen2.5-7B-Instruct solver. For SWE, PROPEL increases the share of generations at the targeted solve rate from $9.8\% \rightarrow 19.6\%$ for Qwen3.5-27B on repositories not seen during training of probe and generator.

Problem

• task generation
• reinforcement learning
• learnable frontier
• solver bottleneck
• software engineering tasks

Innovation

• task generation
• learnable frontier
• solver-amortized training
• activation probe
• reinforcement learning