Better, Faster: Harnessing Self-Improvement in Large Reasoning Models

2026-05-24

AI Summary

This work addresses the vulnerability of large reasoning models to performance degradation or collapse during unsupervised self-improvement training, primarily caused by data imbalance and overthinking. To mitigate these issues, the authors propose the HSIR framework together with the H-GRPO algorithm, which employs verification-guided early-exit sampling to alleviate data imbalance and introduces, for the first time, an intrinsic diversity score as an extrinsic reward in reinforcement learning. This approach jointly optimizes reasoning accuracy, conciseness, and path diversity. Experimental results demonstrate that the method achieves up to a 10.9% average performance gain across multiple complex reasoning tasks while reducing reasoning overhead by as much as 42.4%.

Abstract

Self-improvement training enables the large reasoning models (LRMs) to improve themselves by self-generating reasoning trajectories as training data without external supervision. However, we find that this method often falls short in complex reasoning tasks and even leads to model collapse. Through a series of preliminary analyses, we reveal two problems: (1) data imbalance, where most training samples are simple, but the challenging yet crucial samples are scarce; (2) overthinking, where many undesired samples with redundant reasoning steps are used for self-training. To this end, we propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches. Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance by efficiently collecting more accurate solutions for difficult queries, and designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions. We apply HSIR to various post-training paradigms, among which we further propose H-GRPO, an enhanced GRPO algorithm that leverages the intrinsic diversity as an external reward to encourage concise and diverse reasoning via reinforcement learning. Extensive results show that HSIR not only effectively enhances the reasoning performance, i.e., bringing up to +10.9% average performance gains, but also significantly improves the reasoning efficiency by reducing up to 42.4% relative inference overhead.

Problem

• data imbalance
• overthinking
• self-improvement
• large reasoning models
• reasoning efficiency

Innovation

• self-improvement
• reasoning efficiency
• data imbalance
• intrinsic diversity
• reinforcement learning