Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

2025-08-19

AI Summary

Reinforcement Learning from Verification Results (RLVR) improves large language models’ (LLMs) Pass@1 performance on complex reasoning tasks but reduces policy entropy and output diversity, thereby limiting Pass@k—reflecting the model’s upper bound on reasoning capability. Method: We propose Dynamic Problem Augmentation (DPA), a self-play–based variant problem synthesis (SvS) framework integrated into RLVR. DPA leverages the model’s own correct solutions to generate semantically equivalent yet syntactically novel problems, enabling self-evolution during training while stabilizing policy entropy. It combines online self-play with variational problem synthesis to ensure answer consistency of augmented problems. Contribution/Results: DPA is scalable across model sizes—from medium-scale to 32B parameters—and demonstrates robust generalization. On AIME24 and AIME25, it improves Pass@32 by 18.3% and 22.8%, respectively, and achieves consistent gains across 12 diverse reasoning benchmarks, validating its effectiveness, robustness, and broad applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, vanilla RLVR training has been shown to improve Pass@1 performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass@k performance, which typically represents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's generation diversity from the perspective of training problems and find that augmenting and updating training problems helps mitigate entropy collapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively maintains policy entropy during training and substantially improves Pass@k compared with standard RLVR, sustaining prolonged improvements and achieving absolute gains of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and AIME25 benchmarks. Experiments on 12 reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS.

Problem

• Addresses policy entropy collapse in RLVR training
• Enhances generation diversity for improved Pass@k performance
• Synthesizes variational problems to sustain training improvements

Innovation

• Self-play with variational problem synthesis
• Augmenting training problems to maintain entropy
• Using correct solutions to synthesize identical-answer problems