Parameter Exploration for RLVR via Variational Learning

2026-08-10

AI Summary

This work addresses a key limitation in current reinforcement learning approaches for large language models, which predominantly rely on action-space exploration—such as temperature scaling—and struggle to effectively reorder tokens, often leading to training divergence or stagnation. To overcome this, the paper introduces Perturbed Parameter Policy Optimization (3PO), the first systematic framework leveraging parameter-space exploration. Built upon a variational formulation of the policy posterior, 3PO generates diverse trajectories through parameter perturbations and enhances exploration efficiency via a reward-based grouping mechanism. Evaluated on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks, 3PO consistently outperforms standard GRPO, yielding substantial gains in downstream performance with negligible computational overhead while significantly reducing zero-advantage groups and erroneous outputs.

Abstract

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.

Problem

• exploration
• reinforcement learning
• large language models
• parameter-space
• action-space

Innovation

• parameter-space exploration
• reinforcement learning for LLMs
• Perturbed Parameter Policy Optimization
• rollout diversity
• posterior sampling