TREK: Distill to Explore, Reinforce to Refine

2026-07-06

AI Summary

This work addresses the stagnation of reinforcement learning policies under challenging prompts due to a lack of effective reasoning trajectories. To overcome this, the authors propose TREK, a novel approach that first leverages a teacher model—either black-box or white-box—to generate and filter high-value candidate solutions, thereby expanding the exploration support set for the student policy. These distilled modes are then injected into the student via forward KL divergence alignment, followed by refinement through Group Relative Policy Optimization (GRPO). Crucially, TREK innovatively employs knowledge distillation to enhance exploration rather than mere imitation and efficiently identifies hard examples requiring consolidation. Experiments demonstrate that TREK substantially improves optimization efficiency, boosting the average AIME 2024/2025 score by over 3 points on Qwen3-8B, achieving success rates of 82.8% and 26.7% on ALFWorld and ScienceWorld, respectively, and significantly accelerating convergence on the most difficult tasks.

Abstract

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-$r$ proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.

Problem

• hard prompts
• policy support
• reasoning trajectories
• reinforcement learning
• exploration

Innovation

• TREK
• forward KL
• policy support expansion
• distillation for exploration
• GRPO