Boosting LLM Exploration via Weak-Model Guidance in RLVR

2026-08-27

AI Summary

本文提出利用弱模型引导的方法解决RLVR中因奖励验证导致的策略熵下降问题，通过引入外部前缀促进探索，提高推理覆盖率。

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Problem

• Reinforcement Learning
• Verifiable Rewards
• Policy Entropy
• Reasoning Coverage
• Entropy Collapse

Innovation

• Weak-Model Guidance
• Generative Diversity
• Distributional Discrepancy
• Entropy Preservation
• RLVR