How Far Can Unsupervised RLVR Scale LLM Training?

2026-03-09

AI Summary

This study investigates the efficacy and limitations of unsupervised reinforcement learning with verifiable rewards (URLVR) in training large language models. Addressing the challenge of constructing reward signals without ground-truth labels, the work proposes a unified theoretical framework that reveals intrinsic reward mechanisms as essentially sharpening the initial distribution. It further demonstrates that performance hinges on the alignment between the model’s initial confidence and correctness. The paper introduces “model collapse step” as a practical metric to assess the compatibility between a model’s prior and its trainability under reinforcement learning, observing a common performance trend of initial improvement followed by degradation across intrinsic URLVR methods. Additionally, it explores an extrinsic reward approach grounded in computational asymmetry, providing preliminary evidence of its potential to surpass the confidence-correctness performance ceiling inherent to intrinsic methods.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Reinforcement Learning
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

Problem

• Unsupervised Reinforcement Learning
• Verifiable Rewards
• Large Language Models
• Model Collapse
• Reward Scaling

Innovation

• unsupervised reinforcement learning
• verifiable rewards
• model collapse
• reward sharpening
• computational asymmetry