Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement

2025-11-20

AI Summary

Large reasoning models (LRMs) trained via reinforcement learning with outcome-correctness rewards often suffer from poor internal reasoning quality—manifesting as overthinking, underthinking, redundancy, and disordered reasoning. To address this, we propose a selective self-rewriting framework that integrates self-rewriting with GRPO-based reinforcement learning for the first time. Leveraging LLM-based self-reward signals, the model autonomously rewrites only the reasoning chains of simple samples; rewritten and original outputs are jointly processed within a single batch, incurring only ~10% additional computational overhead. Our method preserves final answer accuracy while substantially improving reasoning process quality: answer accuracy increases by 0.6 points, average reasoning length decreases by 46%, and internal reasoning quality—as evaluated by LLM-as-a-judge—improves by 7.2 points. It effectively mitigates all four major reasoning deficiencies.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Through reinforcement learning (RL) with outcome correctness rewards, large reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal reasoning process. This deficiency leads to suboptimal internal reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own reasoning texts, and subsequently learns from the rewritten reasoning to improve the internal thought process quality. For algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL algorithm and introducing only ~10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter reasoning (-46%) even without explicit instructions in rewriting prompts to reduce reasoning length, outperforming existing strong baselines. In terms of internal reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal reasoning flaws.

Problem

• Improving internal reasoning quality in large language models
• Addressing suboptimal thinking patterns like over-thinking and redundancy
• Enhancing accuracy while reducing reasoning length through self-rewriting

Innovation

• Self-rewriting framework improves internal reasoning quality
• Selective rewriting preserves original reinforcement learning signals
• Combined rewriting and generation in single batch minimizes overhead