When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

2026-03-17

AI Summary

Unsupervised reinforcement learning (RL) often fails in mathematical reasoning due to opaque training dynamics, policy collapse, and reward hacking, with little systematic understanding of the conditions under which it succeeds. This work proposes a concise, deterministic intrinsic reward mechanism integrated with large language models for unsupervised RL and introduces a geometric perspective based on manifold envelopes to analyze training dynamics. We reveal, for the first time, that a model’s prior logical capabilities are the critical determinant of success or failure in unsupervised RL. Furthermore, we establish low-dimensional manifold envelopes as an interpretable geometric criterion for training stability, thereby delineating the effective boundary within which this approach can reliably enhance mathematical reasoning.

Technology Category

• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Although outcome-based reinforcement learning (RL) significantly advances the mathematical reasoning capabilities of Large Language Models (LLMs), its reliance on computationally expensive ground-truth annotations imposes a severe scalability bottleneck. Unsupervised RL guided by intrinsic rewards offers a scalable alternative, yet it suffers from opaque training dynamics and catastrophic instability, such as policy collapse and reward hacking. In this paper, we first design and evaluate a suite of intrinsic rewards that explicitly enforce concise and certain generation. Second, to discover the boundaries of this approach, we test base models across a spectrum of intrinsic reasoning capabilities, revealing how a model's foundational logical prior dictates its success or failure. Finally, to demystify why certain configurations stabilize while others collapse, we introduce a novel geometric diagnostic lens, showing that successful cases are enveloped by manifolds. Ultimately, our work goes beyond merely demonstrating that enforcing concise and certain responses successfully boosts mathematical reasoning; we reveal when this unsupervised approach breaks down and geometrically diagnose why.

Problem

• Unsupervised RL
• Mathematical Reasoning
• Intrinsic Rewards
• Policy Collapse
• Reward Hacking

Innovation

• unsupervised reinforcement learning
• intrinsic reward
• mathematical reasoning
• manifold envelopment
• logical prior