ExpRL: Exploratory RL for LLM Mid-Training

2026-06-15

AI Summary

This work addresses the limitations of traditional approaches that rely on handcrafted reasoning skills and sparse-reward reinforcement learning, which struggle with the diverse, compositional reasoning required for complex problems and are constrained by the coverage of the base model. The authors propose ExpRL, a method that integrates an automated exploration mechanism based on reinforcement learning into large language model training. ExpRL constructs problem-specific dense rewards—derived from human question-answering data at either the process or outcome level—and leverages a large language model as a critic to generate implicit reward signals, guiding the acquisition of effective intermediate reasoning behaviors. Crucially, it eliminates the need for manual skill specification and enables reinforcement of partial progress, thereby overcoming the constraints of supervised fine-tuning and sparse-reward RL. Experiments demonstrate that ExpRL significantly outperforms existing methods on mathematical reasoning tasks and exhibits strong cross-domain generalization.

Abstract

Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.

Problem

• sparse reward reinforcement learning
• mid-training
• reasoning skills
• large language models
• reward scaffolding

Innovation

• Exploratory RL
• reward scaffolds
• dense reward
• mid-training
• reasoning traces